00000100:00100000:0.0:1789760688.566978:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484104064 00000100:00100000:0.0:1789760688.566997:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484104064 00000100:00100000:0.0:1789760688.567100:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 677802449519 time 677842114691 state 0x0 ref 1 00000100:00100000:0.0:1789760688.567116:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 677842375107 time 677842114691 state 0x1 ref 1 00000100:00100000:0.0:1789760688.567145:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 677842375107 time 677898076196 state 0x0 ref 1 00000100:00100000:0.0:1789760688.567160:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484104064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760688.622704:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17465344 -> 17469440 00010000:00010000:0.0:1789760688.622729:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396206e55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17465344->17469439] (req 17465344->17469439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.622743:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396206e55 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17465344->17469439] (req 17465344->17469439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.622752:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396206e55 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17465344->17469439] (req 17465344->17469439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.622760:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760688.622765:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396206e55 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17465344->17469439] (req 17465344->17469439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.622774:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396206e55 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17465344->17469439] (req 17465344->17469439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.622793:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396206e55 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17465344->17469439] (req 17465344->17469439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.622874:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396206e55 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17465344->17469439] (req 17465344->17469439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760688.623001:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484104064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55840us (56196us total) trans 4294971582 rc 0/0 00000100:00100000:0.0:1789760688.623011:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4293 00000100:00100000:0.0:1789760688.623020:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 677842375107 time 677898076196 state 0x2 ref 0 00002000:00010000:0.0:1789760688.682390:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17469440 -> 17473536 00010000:00010000:0.0:1789760688.682412:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206e5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17469440->17473535] (req 17469440->17473535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.682425:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206e5c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17469440->17473535] (req 17469440->17473535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.682433:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206e5c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17469440->17473535] (req 17469440->17473535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.682440:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760688.682444:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206e5c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17469440->17473535] (req 17469440->17473535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.682453:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206e5c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17469440->17473535] (req 17469440->17473535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.682479:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206e5c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17469440->17473535] (req 17469440->17473535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.682487:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206e5c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17469440->17473535] (req 17469440->17473535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760688.683366:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484104192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56635us (56974us total) trans 4294971583 rc 0/0 00000100:00100000:0.0:1789760688.683378:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4294 00000100:00100000:0.0:1789760688.683388:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 677898336612 time 677957635194 state 0x2 ref 0 00002000:00010000:0.0:1789760688.759312:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17473536 -> 17477632 00010000:00010000:0.0:1789760688.759328:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396206e63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17473536->17477631] (req 17473536->17477631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.759341:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396206e63 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17473536->17477631] (req 17473536->17477631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.759350:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396206e63 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17473536->17477631] (req 17473536->17477631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.759358:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760688.759363:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396206e63 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17473536->17477631] (req 17473536->17477631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.759378:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396206e63 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17473536->17477631] (req 17473536->17477631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.759398:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396206e63 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17473536->17477631] (req 17473536->17477631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.759405:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396206e63 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17473536->17477631] (req 17473536->17477631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760688.760720:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484104320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69967us (70607us total) trans 4294971584 rc 0/0 00000100:00100000:0.0:1789760688.760741:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4295 00000100:00100000:0.0:1789760688.760752:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 677957895610 time 678020444225 state 0x2 ref 0 00000100:00100000:0.0:1789760688.764663:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484104448 00000100:00100000:0.0:1789760688.764672:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484104448 00000100:00100000:0.0:1789760688.764859:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 677957895610 time 678020444225 state 0x0 ref 1 00000100:00100000:0.0:1789760688.764874:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678020704641 time 678020444225 state 0x1 ref 1 00002000:00010000:0.0:1789760688.830639:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17477632 -> 17481728 00010000:00010000:0.0:1789760688.830654:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396206e6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17477632->17481727] (req 17477632->17481727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.830667:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396206e6a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17477632->17481727] (req 17477632->17481727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.830680:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396206e6a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17477632->17481727] (req 17477632->17481727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.830687:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760688.830692:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396206e6a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17477632->17481727] (req 17477632->17481727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.830701:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396206e6a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17477632->17481727] (req 17477632->17481727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.830732:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396206e6a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17477632->17481727] (req 17477632->17481727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.830740:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396206e6a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17477632->17481727] (req 17477632->17481727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760688.830837:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484104448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65842us (66316us total) trans 4294971585 rc 0/0 00000100:00100000:0.0:1789760688.830848:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4296 00000100:00100000:0.0:1789760688.830859:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678020704641 time 678095906063 state 0x2 ref 0 00000100:00100000:0.0:1789760688.836556:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484104576 00000100:00100000:0.0:1789760688.836566:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484104576 00000100:00100000:0.0:1789760688.836625:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678020704641 time 678095906063 state 0x0 ref 1 00000100:00100000:0.0:1789760688.836641:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678096166479 time 678095906063 state 0x1 ref 1 00000100:00100000:0.0:1789760688.836673:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678096166479 time 678167603751 state 0x0 ref 1 00000100:00100000:0.0:1789760688.836686:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484104576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760688.836742:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17481728, mtime: 0x6aad94b0, atime: 0x0, ctime: 0x6aad94b0, blocks: 0x8807 00010000:00010000:0.0:1789760688.836752:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206e71 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.836769:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=678s v=(0 0) 00010000:00010000:0.0:1789760688.836773:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760688.836780:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206e71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17481728->17485823] (req 17481728->17485823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.836794:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206e71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17481728->17485823] (req 17481728->17485823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760688.909207:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971586 is committed 00000100:00100000:0.0:1789760688.919228:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484104704 00000100:00100000:0.0:1789760688.919238:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484104704 00000100:00100000:0.0:1789760688.919302:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678096166479 time 678167603751 state 0x0 ref 1 00000100:00100000:0.0:1789760688.919317:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678167864167 time 678167603751 state 0x1 ref 1 00000100:00100000:0.0:1789760688.919347:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678167864167 time 678250277803 state 0x0 ref 1 00000100:00100000:0.0:1789760688.919360:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b82300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484104704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760688.919410:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17485824, mtime: 0x6aad94b0, atime: 0x0, ctime: 0x6aad94b0, blocks: 0x8807 00010000:00010000:0.0:1789760688.919419:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206e78 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.919435:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=678s v=(0 0) 00010000:00010000:0.0:1789760688.919440:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760688.919447:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206e78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17485824->17489919] (req 17485824->17489919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760688.919461:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206e78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17485824->17489919] (req 17485824->17489919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760688.961507:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971587 is committed 00010000:00010000:0.0:1789760689.036220:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760689.275597:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484105344 00000100:00100000:0.0:1789760689.275608:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484105344 00000100:00100000:0.0:1789760689.275666:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678439404295 time 678552845412 state 0x0 ref 1 00000100:00100000:0.0:1789760689.275681:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678553105828 time 678552845412 state 0x1 ref 1 00000100:00100000:0.0:1789760689.275727:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678553105828 time 678606656971 state 0x0 ref 1 00000100:00100000:0.0:1789760689.275742:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484105344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760689.275791:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17506304, mtime: 0x6aad94b1, atime: 0x0, ctime: 0x6aad94b1, blocks: 0x8807 00010000:00010000:0.0:1789760689.275800:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.275823:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=678s v=(0 0) 00010000:00010000:0.0:1789760689.275827:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760689.275835:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17506304->17510399] (req 17506304->17510399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.275849:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17506304->17510399] (req 17506304->17510399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760689.311657:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17506304 -> 17510400 00010000:00010000:0.0:1789760689.311671:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17506304->17510399] (req 17506304->17510399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.311684:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17506304->17510399] (req 17506304->17510399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.311693:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17506304->17510399] (req 17506304->17510399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.311699:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760689.311703:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17506304->17510399] (req 17506304->17510399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.311723:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17506304->17510399] (req 17506304->17510399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.311741:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17506304->17510399] (req 17506304->17510399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.311749:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206e9b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17506304->17510399] (req 17506304->17510399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760689.311840:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484105344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36098us (36351us total) trans 4294971592 rc 0/0 00000100:00100000:0.0:1789760689.311850:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4303 00000100:00100000:0.0:1789760689.311858:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678553105828 time 678606656971 state 0x2 ref 0 00002000:00010000:0.0:1789760689.341503:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17510400 -> 17514496 00010000:00010000:0.0:1789760689.341516:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ea2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17510400->17514495] (req 17510400->17514495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.341525:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ea2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17510400->17514495] (req 17510400->17514495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.341531:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ea2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17510400->17514495] (req 17510400->17514495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.341536:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760689.341539:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ea2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17510400->17514495] (req 17510400->17514495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.341545:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ea2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17510400->17514495] (req 17510400->17514495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.341564:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ea2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17510400->17514495] (req 17510400->17514495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.341569:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ea2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17510400->17514495] (req 17510400->17514495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760689.341650:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484105472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 22456us (25878us total) trans 4294971593 rc 0/0 00000100:00100000:0.0:1789760689.341657:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4304 00000100:00100000:0.0:1789760689.341665:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678606917387 time 678650107189 state 0x2 ref 0 00000100:00100000:0.0:1789760689.371763:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484105728 00000100:00100000:0.0:1789760689.371773:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484105728 00000100:00100000:0.0:1789760689.371831:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678650367605 time 678675095762 state 0x0 ref 1 00000100:00100000:0.0:1789760689.371846:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678675356178 time 678675095762 state 0x1 ref 1 00000100:00100000:0.0:1789760689.371879:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678675356178 time 678702810513 state 0x0 ref 1 00000100:00100000:0.0:1789760689.371894:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484105728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760689.371940:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17518592, mtime: 0x6aad94b1, atime: 0x0, ctime: 0x6aad94b1, blocks: 0x8807 00010000:00010000:0.0:1789760689.371948:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.371963:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=678s v=(0 0) 00010000:00010000:0.0:1789760689.371968:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760689.371974:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17518592->17522687] (req 17518592->17522687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.371993:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17518592->17522687] (req 17518592->17522687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760689.409039:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971595 is committed 00002000:00010000:0.0:1789760689.409147:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17518592 -> 17522688 00010000:00010000:0.0:1789760689.409162:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17518592->17522687] (req 17518592->17522687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.409175:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17518592->17522687] (req 17518592->17522687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.409184:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17518592->17522687] (req 17518592->17522687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.409192:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760689.409197:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17518592->17522687] (req 17518592->17522687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.409207:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17518592->17522687] (req 17518592->17522687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.409227:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17518592->17522687] (req 17518592->17522687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.409242:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206eb0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17518592->17522687] (req 17518592->17522687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760689.409339:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484105728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37442us (37960us total) trans 4294971595 rc 0/0 00000100:00100000:0.0:1789760689.409350:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4306 00000100:00100000:0.0:1789760689.409361:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678675356178 time 678702810513 state 0x2 ref 0 00000100:00100000:0.0:1789760689.412742:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484105856 00000100:00100000:0.0:1789760689.412751:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484105856 00000100:00100000:0.0:1789760689.412813:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678675356178 time 678702810513 state 0x0 ref 1 00000100:00100000:0.0:1789760689.412829:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678703070929 time 678702810513 state 0x1 ref 1 00000100:00100000:0.0:1789760689.412908:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678703070929 time 678743837406 state 0x0 ref 1 00000100:00100000:0.0:1789760689.412924:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484105856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760689.412975:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17522688, mtime: 0x6aad94b1, atime: 0x0, ctime: 0x6aad94b1, blocks: 0x8807 00010000:00010000:0.0:1789760689.412984:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206eb7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.413000:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=678s v=(0 0) 00010000:00010000:0.0:1789760689.413005:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760689.413011:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206eb7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17522688->17526783] (req 17522688->17526783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.413031:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206eb7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17522688->17526783] (req 17522688->17526783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760689.488393:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760689.595108:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971597 is committed 00000001:00080000:0.0:1789760689.672425:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971598 is committed 00002000:00010000:0.0:1789760689.672496:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17530880 -> 17534976 00010000:00010000:0.0:1789760689.672513:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396206ec5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17530880->17534975] (req 17530880->17534975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.672527:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396206ec5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17530880->17534975] (req 17530880->17534975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.672538:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396206ec5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17530880->17534975] (req 17530880->17534975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.672546:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760689.672550:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396206ec5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17530880->17534975] (req 17530880->17534975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.672559:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396206ec5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17530880->17534975] (req 17530880->17534975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.672591:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396206ec5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17530880->17534975] (req 17530880->17534975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.672606:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396206ec5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17530880->17534975] (req 17530880->17534975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760689.672698:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484106112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68930us (69918us total) trans 4294971598 rc 0/0 00000100:00100000:0.0:1789760689.672717:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4309 00000100:00100000:0.0:1789760689.672726:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678820069639 time 678934684666 state 0x2 ref 0 00000100:00100000:0.0:1789760689.680003:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484106240 00000100:00100000:0.0:1789760689.680014:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484106240 00000100:00100000:0.0:1789760689.680095:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678820069639 time 678934684666 state 0x0 ref 1 00000100:00100000:0.0:1789760689.680112:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678934945082 time 678934684666 state 0x1 ref 1 00000100:00100000:0.0:1789760689.680145:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678934945082 time 679011075848 state 0x0 ref 1 00000100:00100000:0.0:1789760689.680160:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484106240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760689.680211:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17534976, mtime: 0x6aad94b1, atime: 0x0, ctime: 0x6aad94b1, blocks: 0x8807 00010000:00010000:0.0:1789760689.680220:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.680236:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=679s v=(0 0) 00010000:00010000:0.0:1789760689.680240:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760689.680246:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17534976->17539071] (req 17534976->17539071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.680373:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17534976->17539071] (req 17534976->17539071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760689.772456:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17534976 -> 17539072 00010000:00010000:0.0:1789760689.772481:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17534976->17539071] (req 17534976->17539071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.772680:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17534976->17539071] (req 17534976->17539071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.772691:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17534976->17539071] (req 17534976->17539071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.772709:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760689.772714:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17534976->17539071] (req 17534976->17539071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.772724:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17534976->17539071] (req 17534976->17539071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.772748:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17534976->17539071] (req 17534976->17539071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.772756:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206ecc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17534976->17539071] (req 17534976->17539071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760689.772894:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484106240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 92730us (93235us total) trans 4294971599 rc 0/0 00000100:00100000:0.0:1789760689.772906:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4310 00000100:00100000:0.0:1789760689.772918:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678934945082 time 679011075848 state 0x2 ref 0 00000001:00080000:0.0:1789760689.772972:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971599 is committed 00000100:00100000:0.0:1789760689.776312:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484106368 00000100:00100000:0.0:1789760689.776325:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484106368 00000100:00100000:0.0:1789760689.776408:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 678934945082 time 679011075848 state 0x0 ref 1 00000100:00100000:0.0:1789760689.776428:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679011336264 time 679011075848 state 0x1 ref 1 00000100:00100000:0.0:1789760689.776467:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679011336264 time 679107397563 state 0x0 ref 1 00000100:00100000:0.0:1789760689.776482:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484106368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760689.776548:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17539072, mtime: 0x6aad94b1, atime: 0x0, ctime: 0x6aad94b1, blocks: 0x8807 00010000:00010000:0.0:1789760689.776559:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396206ed3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.776579:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=679s v=(0 0) 00010000:00010000:0.0:1789760689.776584:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760689.776591:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396206ed3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17539072->17543167] (req 17539072->17543167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.776607:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396206ed3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17539072->17543167] (req 17539072->17543167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760689.934541:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17543168 -> 17547264 00010000:00010000:0.0:1789760689.934564:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396206eda lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17543168->17547263] (req 17543168->17547263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.934580:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396206eda lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17543168->17547263] (req 17543168->17547263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.934591:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396206eda lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17543168->17547263] (req 17543168->17547263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.934599:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760689.934605:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396206eda lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17543168->17547263] (req 17543168->17547263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.934615:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396206eda lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17543168->17547263] (req 17543168->17547263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.934638:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396206eda lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17543168->17547263] (req 17543168->17547263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760689.934648:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396206eda lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17543168->17547263] (req 17543168->17547263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760689.934777:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484106496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69388us (70338us total) trans 4294971601 rc 0/0 00000100:00100000:0.0:1789760689.934792:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4312 00000100:00100000:0.0:1789760689.934804:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679107657979 time 679196300169 state 0x2 ref 0 00000100:00100000:0.0:1789760689.994010:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:0.0:1789760690.075412:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17551360 -> 17555456 00010000:00010000:0.0:1789760690.075433:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ee8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17551360->17555455] (req 17551360->17555455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.075448:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ee8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17551360->17555455] (req 17551360->17555455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.075458:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ee8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17551360->17555455] (req 17551360->17555455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.075467:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760690.075473:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ee8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17551360->17555455] (req 17551360->17555455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.075484:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ee8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17551360->17555455] (req 17551360->17555455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.075510:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ee8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17551360->17555455] (req 17551360->17555455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.075518:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396206ee8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17551360->17555455] (req 17551360->17555455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760690.075692:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484106752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 81021us (81689us total) trans 4294971603 rc 0/0 00000100:00100000:0.0:1789760690.075712:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4314 00000100:00100000:0.0:1789760690.075725:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679273369918 time 679325579063 state 0x2 ref 0 00000100:00100000:0.0:1789760690.079323:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:0.0:1789760690.145461:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17555456 -> 17559552 00010000:00010000:0.0:1789760690.145499:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396206eef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17555456->17559551] (req 17555456->17559551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.145516:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396206eef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17555456->17559551] (req 17555456->17559551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.145527:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396206eef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17555456->17559551] (req 17555456->17559551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.145536:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760690.145542:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396206eef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17555456->17559551] (req 17555456->17559551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.145554:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396206eef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17555456->17559551] (req 17555456->17559551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.145581:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396206eef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17555456->17559551] (req 17555456->17559551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.145596:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396206eef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17555456->17559551] (req 17555456->17559551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760690.145726:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484106880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65745us (66411us total) trans 4294971604 rc 0/0 00000100:00100000:0.0:1789760690.145740:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4315 00000100:00100000:0.0:1789760690.145753:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679325839479 time 679410738344 state 0x2 ref 0 00000100:00100000:0.0:1789760690.148924:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484107008 00000100:00100000:0.0:1789760690.148938:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484107008 00000100:00100000:0.0:1789760690.149027:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679325839479 time 679410738344 state 0x0 ref 1 00000100:00100000:0.0:1789760690.149048:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679410998760 time 679410738344 state 0x1 ref 1 00000100:00100000:0.0:1789760690.149120:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679410998760 time 679480049777 state 0x0 ref 1 00000100:00100000:0.0:1789760690.149136:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b82a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484107008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760690.149201:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17559552, mtime: 0x6aad94b1, atime: 0x0, ctime: 0x6aad94b1, blocks: 0x8807 00010000:00010000:0.0:1789760690.149211:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.149234:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=679s v=(0 0) 00010000:00010000:0.0:1789760690.149239:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760690.149247:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17559552->17563647] (req 17559552->17563647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.149269:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17559552->17563647] (req 17559552->17563647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760690.198359:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17559552 -> 17563648 00002000:00010000:0.0:1789760690.198365:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760689 -> 1789760690 00002000:00010000:0.0:1789760690.198369:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760689 -> 1789760690 00010000:00010000:0.0:1789760690.198397:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17559552->17563647] (req 17559552->17563647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.198413:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17559552->17563647] (req 17559552->17563647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.198424:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17559552->17563647] (req 17559552->17563647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.198434:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760690.198440:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17559552->17563647] (req 17559552->17563647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.198450:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17559552->17563647] (req 17559552->17563647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.198476:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17559552->17563647] (req 17559552->17563647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.198599:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206ef6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17559552->17563647] (req 17559552->17563647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760690.198757:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b82a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484107008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49613us (49946us total) trans 4294971605 rc 0/0 00000100:00100000:0.0:1789760690.198770:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4316 00000100:00100000:0.0:1789760690.198783:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679410998760 time 679480049777 state 0x2 ref 0 00000100:00100000:0.0:1789760690.203590:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484107136 00000100:00100000:0.0:1789760690.203604:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484107136 00000100:00100000:0.0:1789760690.203687:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679410998760 time 679480049777 state 0x0 ref 1 00000100:00100000:0.0:1789760690.203718:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679480310193 time 679480049777 state 0x1 ref 1 00000100:00100000:0.0:1789760690.203752:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679480310193 time 679534682137 state 0x0 ref 1 00000100:00100000:0.0:1789760690.203769:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484107136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760690.203964:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17563648, mtime: 0x6aad94b2, atime: 0x0, ctime: 0x6aad94b2, blocks: 0x8807 00010000:00010000:0.0:1789760690.203976:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396206efd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.203997:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=679s v=(0 0) 00010000:00010000:0.0:1789760690.204002:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760690.204010:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396206efd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17563648->17567743] (req 17563648->17567743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.204035:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396206efd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17563648->17567743] (req 17563648->17567743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760690.292321:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17567744 -> 17571840 00010000:00010000:0.0:1789760690.292341:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396206f04 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17567744->17571839] (req 17567744->17571839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.292358:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396206f04 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17567744->17571839] (req 17567744->17571839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.292369:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396206f04 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17567744->17571839] (req 17567744->17571839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.292378:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760690.292384:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396206f04 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17567744->17571839] (req 17567744->17571839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.292394:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396206f04 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17567744->17571839] (req 17567744->17571839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.292421:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396206f04 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17567744->17571839] (req 17567744->17571839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.292430:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396206f04 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17567744->17571839] (req 17567744->17571839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760690.292670:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484107264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38879us (39639us total) trans 4294971607 rc 0/0 00000100:00100000:0.0:1789760690.292683:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4318 00000100:00100000:0.0:1789760690.292695:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679534942553 time 679584700446 state 0x2 ref 0 00000100:00100000:0.0:1789760690.296051:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484107392 00000100:00100000:0.0:1789760690.296100:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484107392 00000100:00100000:0.0:1789760690.296183:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679534942553 time 679584700446 state 0x0 ref 1 00000100:00100000:0.0:1789760690.296203:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679584960862 time 679584700446 state 0x1 ref 1 00000100:00100000:0.0:1789760690.296228:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679584960862 time 679627158573 state 0x0 ref 1 00000100:00100000:0.0:1789760690.296243:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484107392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760690.296308:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17571840, mtime: 0x6aad94b2, atime: 0x0, ctime: 0x6aad94b2, blocks: 0x8807 00010000:00010000:0.0:1789760690.296319:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.296340:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=679s v=(0 0) 00010000:00010000:0.0:1789760690.296345:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760690.296353:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17571840->17575935] (req 17571840->17575935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.296369:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17571840->17575935] (req 17571840->17575935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760690.363766:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17571840 -> 17575936 00010000:00010000:0.0:1789760690.363787:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17571840->17575935] (req 17571840->17575935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.363804:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17571840->17575935] (req 17571840->17575935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.363813:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17571840->17575935] (req 17571840->17575935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.363820:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760690.363826:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17571840->17575935] (req 17571840->17575935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.363836:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17571840->17575935] (req 17571840->17575935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.363859:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17571840->17575935] (req 17571840->17575935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.363867:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f0b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17571840->17575935] (req 17571840->17575935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760690.364002:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484107392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67757us (68040us total) trans 4294971608 rc 0/0 00000100:00100000:0.0:1789760690.364018:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4319 00000100:00100000:0.0:1789760690.364030:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679584960862 time 679627158573 state 0x2 ref 0 00000100:00100000:0.0:1789760690.372593:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484107520 00000100:00100000:0.0:1789760690.372606:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484107520 00000100:00100000:0.0:1789760690.372686:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679584960862 time 679627158573 state 0x0 ref 1 00000100:00100000:0.0:1789760690.372714:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679627418989 time 679627158573 state 0x1 ref 1 00000100:00100000:0.0:1789760690.372810:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679627418989 time 679703739131 state 0x0 ref 1 00000100:00100000:0.0:1789760690.372828:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484107520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760690.372895:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17575936, mtime: 0x6aad94b2, atime: 0x0, ctime: 0x6aad94b2, blocks: 0x8807 00010000:00010000:0.0:1789760690.372906:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f12 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.372926:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=679s v=(0 0) 00010000:00010000:0.0:1789760690.372932:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760690.372940:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f12 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17575936->17580031] (req 17575936->17580031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.372956:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f12 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17575936->17580031] (req 17575936->17580031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760690.576509:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df02f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508897280:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760690.576572:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:0.0:1789760690.583770:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:0.0:1789760690.583796:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:0.0:1789760690.583801:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:0.0:1789760690.583847:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b18700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508897408:0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.583977:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760690.584046:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b18000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508897920:0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584098:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760690.584123:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b18e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508898432:0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584154:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760690.584185:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [2->3] 00000100:00100000:0.0:1789760690.584192:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b19c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508897792:0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584221:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760690.584233:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508898304:0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584256:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760690.584332:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df02f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508897280:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760690.584369:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508898688 00000100:00100000:0.0:1789760690.584378:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508898688 00000100:00100000:0.0:1789760690.584442:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508897792 00000100:00100000:0.0:1789760690.584446:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508897792 00000100:00100000:0.0:1789760690.584467:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 815 00000100:00100000:0.0:1789760690.584476:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508897536:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584560:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9473800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508897536:12345-0@lo:400:kworker.0 Request processed in 84us (7451us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760690.584569:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 815 00000100:00100000:0.0:1789760690.584591:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 816 00000100:00100000:0.0:1789760690.584599:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508898688:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584632:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508898688:12345-0@lo:400:kworker.0 Request processed in 34us (7225us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760690.584638:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 816 00000100:00100000:0.0:1789760690.584652:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 817 00000100:00100000:0.0:1789760690.584658:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba2a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508897792:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584690:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba2a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508897792:12345-0@lo:400:kworker.0 Request processed in 32us (468us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760690.584706:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 817 00000100:00100000:0.0:1789760690.584732:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508897664 00000100:00100000:0.0:1789760690.584737:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508897664 00000100:00100000:0.0:1789760690.584770:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508897920 00000100:00100000:0.0:1789760690.584773:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508897920 00000100:00100000:0.0:1789760690.584788:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 399 00000100:00100000:0.0:1789760690.584795:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472300 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508897664:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584832:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472300 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508897664:12345-0@lo:400:kworker.0 Request processed in 37us (7509us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760690.584838:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 399 00000100:00100000:0.0:1789760690.584850:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 400 00000100:00100000:0.0:1789760690.584857:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508897920:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584882:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508897920:12345-0@lo:400:kworker.0 Request processed in 28us (783us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760690.584888:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 400 00000100:00100000:0.0:1789760690.584905:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508897408 00000100:00100000:0.0:1789760690.584909:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508897408 00000100:00100000:0.0:1789760690.584935:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 458 00000100:00100000:0.0:1789760690.584941:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4d2911850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508897408:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.584970:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4d2911850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508897408:12345-0@lo:400:kworker.0 Request processed in 29us (993us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760690.584976:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 458 00000100:00100000:0.0:1789760690.585011:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508898432 00000100:00100000:0.0:1789760690.585015:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508898432 00000100:00100000:0.0:1789760690.585045:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508898304 00000100:00100000:0.0:1789760690.585047:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508898304 00000100:00100000:0.0:1789760690.585090:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 289 00000100:00100000:0.0:1789760690.585098:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508898432:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.585132:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508898432:12345-0@lo:400:kworker.0 Request processed in 36us (977us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760690.585138:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 289 00000100:00100000:0.0:1789760690.585151:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 290 00000100:00100000:0.0:1789760690.585157:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508898304:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.585185:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508898304:12345-0@lo:400:kworker.0 Request processed in 29us (928us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760690.585191:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 290 00000100:00100000:0.0:1789760690.585224:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b19c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508897792:0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.585235:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508898304:0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.585243:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b18700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508897408:0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.585253:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b18000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508897920:0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.585263:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b18e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508898432:0@lo:400:kworker.0 00000100:00080000:0.0:1789760690.585296:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:0.0:1789760690.585300:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:0.0:1789760690.591962:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 819 00000100:00100000:0.0:1789760690.591990:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508898560:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760690.592144:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508898560:12345-0@lo:400:kworker.0 Request processed in 171us (14935us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760690.592153:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 819 00000001:00080000:0.0:1789760690.641767:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971611 is committed 00000001:00080000:0.0:1789760690.707310:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971612 is committed 00002000:00010000:0.0:1789760690.707513:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17588224 -> 17592320 00010000:00010000:0.0:1789760690.707533:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396206f27 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17588224->17592319] (req 17588224->17592319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.707549:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396206f27 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17588224->17592319] (req 17588224->17592319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.707560:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396206f27 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17588224->17592319] (req 17588224->17592319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.707568:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760690.707574:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396206f27 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17588224->17592319] (req 17588224->17592319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.707583:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396206f27 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17588224->17592319] (req 17588224->17592319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.707611:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396206f27 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17588224->17592319] (req 17588224->17592319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.707624:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396206f27 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17588224->17592319] (req 17588224->17592319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760690.707851:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484107904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61034us (61300us total) trans 4294971612 rc 0/0 00000100:00100000:0.0:1789760690.707864:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4323 00000100:00100000:0.0:1789760690.707875:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679871099397 time 679977727152 state 0x2 ref 0 00000100:00100000:0.0:1789760690.716830:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:0.0:1789760690.753114:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17592320 -> 17596416 00010000:00010000:0.0:1789760690.753135:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396206f2e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17592320->17596415] (req 17592320->17596415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.753152:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396206f2e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17592320->17596415] (req 17592320->17596415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.753162:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396206f2e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17592320->17596415] (req 17592320->17596415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.753171:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760690.753176:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396206f2e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17592320->17596415] (req 17592320->17596415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.753186:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396206f2e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17592320->17596415] (req 17592320->17596415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.753224:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396206f2e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17592320->17596415] (req 17592320->17596415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.753233:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396206f2e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17592320->17596415] (req 17592320->17596415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760690.753356:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abfb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484108032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36040us (36537us total) trans 4294971613 rc 0/0 00000100:00100000:0.0:1789760690.753367:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4324 00000100:00100000:0.0:1789760690.753379:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 679977987568 time 680048225555 state 0x2 ref 0 00000001:00080000:0.0:1789760690.834474:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971614 is committed 00000100:00100000:0.0:1789760690.839253:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484108288 00000100:00100000:0.0:1789760690.839266:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484108288 00000100:00100000:0.0:1789760690.839348:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680048485971 time 680099574778 state 0x0 ref 1 00000100:00100000:0.0:1789760690.839367:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680099835194 time 680099574778 state 0x1 ref 1 00000100:00100000:0.0:1789760690.839406:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680099835194 time 680170336510 state 0x0 ref 1 00000100:00100000:0.0:1789760690.839423:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484108288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760690.839676:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17600512, mtime: 0x6aad94b2, atime: 0x0, ctime: 0x6aad94b2, blocks: 0x8807 00010000:00010000:0.0:1789760690.839688:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396206f3c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.839724:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=680s v=(0 0) 00010000:00010000:0.0:1789760690.839729:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760690.839737:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396206f3c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17600512->17604607] (req 17600512->17604607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.839755:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396206f3c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17600512->17604607] (req 17600512->17604607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760690.885470:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971615 is committed 00002000:00010000:0.0:1789760690.922274:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17604608 -> 17608704 00010000:00010000:0.0:1789760690.922295:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206f43 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17604608->17608703] (req 17604608->17608703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.922310:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206f43 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17604608->17608703] (req 17604608->17608703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.922320:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206f43 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17604608->17608703] (req 17604608->17608703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.922328:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760690.922334:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206f43 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17604608->17608703] (req 17604608->17608703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.922346:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206f43 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17604608->17608703] (req 17604608->17608703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.922377:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206f43 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17604608->17608703] (req 17604608->17608703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760690.922385:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396206f43 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17604608->17608703] (req 17604608->17608703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760690.922560:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484108416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32733us (33001us total) trans 4294971616 rc 0/0 00000100:00100000:0.0:1789760690.922572:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4327 00000100:00100000:0.0:1789760690.922585:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680170596926 time 680220737843 state 0x2 ref 0 00000100:00100000:0.0:1789760690.928662:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00010000:00010000:0.0:1789760691.089212:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:0.0:1789760691.253887:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17620992 -> 17625088 00002000:00010000:0.0:1789760691.253893:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760690 -> 1789760691 00002000:00010000:0.0:1789760691.253896:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760690 -> 1789760691 00010000:00010000:0.0:1789760691.253914:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396206f5f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17620992->17625087] (req 17620992->17625087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.253929:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396206f5f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17620992->17625087] (req 17620992->17625087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.253940:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396206f5f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17620992->17625087] (req 17620992->17625087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.253954:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760691.253960:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396206f5f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17620992->17625087] (req 17620992->17625087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.253970:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396206f5f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17620992->17625087] (req 17620992->17625087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.253996:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396206f5f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17620992->17625087] (req 17620992->17625087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.254004:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396206f5f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17620992->17625087] (req 17620992->17625087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760691.254220:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484108928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69807us (70072us total) trans 4294971620 rc 0/0 00000100:00100000:0.0:1789760691.254234:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4331 00000100:00100000:0.0:1789760691.254245:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680424164134 time 680515324719 state 0x2 ref 0 00000001:00080000:0.0:1789760691.350880:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971621 is committed 00000100:00100000:0.0:1789760691.430498:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760691.430548:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760691.430573:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760691.430592:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760691.430608:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760691.430632:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760691.430652:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760691.430672:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484109952 00000100:00100000:0.0:1789760691.430676:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484109952 00000100:00100000:0.0:1789760691.435081:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484110080 00000100:00100000:0.0:1789760691.435103:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484110080 00000100:00100000:0.0:1789760691.435186:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 821 00000100:00100000:0.0:1789760691.435212:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abdf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484109696:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760691.435430:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abdf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484109696:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 220us (4853us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760691.435438:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 821 00000100:00100000:0.0:1789760691.435469:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 822 00000100:00100000:0.0:1789760691.435549:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abc380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484109952:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760691.435580:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abc380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484109952:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 38us (4988us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760691.435587:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 822 00000100:00100000:0.0:1789760691.435601:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 823 00000100:00100000:0.0:1789760691.435608:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abe680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484109568:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760691.435634:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abe680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484109568:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 27us (5025us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760691.435639:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 823 00000100:00100000:0.0:1789760691.435652:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 824 00000100:00100000:0.0:1789760691.435658:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abf480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484110080:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760691.435682:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abf480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484110080:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 24us (5028us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760691.435687:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 824 00000100:00100000:0.0:1789760691.435734:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484109824 00000100:00100000:0.0:1789760691.435739:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484109824 00000100:00100000:0.0:1789760691.436350:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 291 00000100:00100000:0.0:1789760691.436362:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484109824:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760691.436403:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484109824:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 45us (5769us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760691.436410:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 291 00000100:00080000:0.0:1789760691.436434:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:0.0:1789760691.436439:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00002000:00010000:0.0:1789760691.570436:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17633280 -> 17637376 00010000:00010000:0.0:1789760691.570465:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396206f74 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17633280->17637375] (req 17633280->17637375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.570548:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396206f74 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17633280->17637375] (req 17633280->17637375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.570559:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396206f74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17633280->17637375] (req 17633280->17637375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.570567:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760691.570573:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396206f74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17633280->17637375] (req 17633280->17637375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.570584:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396206f74 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17633280->17637375] (req 17633280->17637375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.570612:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396206f74 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17633280->17637375] (req 17633280->17637375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.570622:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396206f74 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17633280->17637375] (req 17633280->17637375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760691.570778:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484109312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 139906us (140287us total) trans 4294971623 rc 0/0 00000100:00100000:0.0:1789760691.570791:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4334 00000100:00100000:0.0:1789760691.570803:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680693554363 time 680761783132 state 0x2 ref 0 00000100:00100000:0.0:1789760691.582123:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484110208 00000100:00100000:0.0:1789760691.582137:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484110208 00000100:00100000:0.0:1789760691.582222:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680693554363 time 680761783132 state 0x0 ref 1 00000100:00100000:0.0:1789760691.582240:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680762043548 time 680761783132 state 0x1 ref 1 00000100:00100000:0.0:1789760691.697840:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484110336 00000100:00100000:0.0:1789760691.697864:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484110336 00000100:00100000:0.0:1789760691.697962:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680762043548 time 680913508947 state 0x0 ref 1 00000100:00100000:0.0:1789760691.697982:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 680913769363 time 680913508947 state 0x1 ref 1 00000100:00100000:0.0:1789760691.803704:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760691.908168:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484110592 00000100:00100000:0.0:1789760691.908181:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484110592 00000100:00100000:0.0:1789760691.908269:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681029243989 time 681134910111 state 0x0 ref 1 00000100:00100000:0.0:1789760691.908288:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681135170527 time 681134910111 state 0x1 ref 1 00000100:00100000:0.0:1789760691.908354:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681135170527 time 681239284610 state 0x0 ref 1 00000100:00100000:0.0:1789760691.908370:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484110592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760691.908479:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17649664, mtime: 0x6aad94b3, atime: 0x0, ctime: 0x6aad94b3, blocks: 0x8807 00010000:00010000:0.0:1789760691.908491:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.908512:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=681s v=(0 0) 00010000:00010000:0.0:1789760691.908517:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760691.908526:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17649664->17653759] (req 17649664->17653759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.908543:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17649664->17653759] (req 17649664->17653759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760691.952320:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17649664 -> 17653760 00010000:00010000:0.0:1789760691.952340:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17649664->17653759] (req 17649664->17653759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.952355:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17649664->17653759] (req 17649664->17653759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.952364:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17649664->17653759] (req 17649664->17653759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.952372:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760691.952377:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17649664->17653759] (req 17649664->17653759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.952387:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17649664->17653759] (req 17649664->17653759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.952410:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17649664->17653759] (req 17649664->17653759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.952418:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f90 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17649664->17653759] (req 17649664->17653759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760691.952550:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484110592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44176us (44660us total) trans 4294971627 rc 0/0 00000100:00100000:0.0:1789760691.952563:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4338 00000100:00100000:0.0:1789760691.952575:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681135170527 time 681239284610 state 0x2 ref 0 00000100:00100000:0.0:1789760691.955570:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484110720 00000100:00100000:0.0:1789760691.955587:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484110720 00000100:00100000:0.0:1789760691.955672:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681135170527 time 681239284610 state 0x0 ref 1 00000100:00100000:0.0:1789760691.955704:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681239545026 time 681239284610 state 0x1 ref 1 00000100:00100000:0.0:1789760691.955783:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681239545026 time 681286713485 state 0x0 ref 1 00000100:00100000:0.0:1789760691.955799:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484110720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760691.955862:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17653760, mtime: 0x6aad94b3, atime: 0x0, ctime: 0x6aad94b3, blocks: 0x8807 00010000:00010000:0.0:1789760691.955874:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396206f97 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.955893:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=681s v=(0 0) 00010000:00010000:0.0:1789760691.955898:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760691.955906:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396206f97 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17653760->17657855] (req 17653760->17657855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760691.955922:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396206f97 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17653760->17657855] (req 17653760->17657855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760692.008378:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681239545026 time 681286713485 state 0x0 ref 1 00000100:00100000:0.0:1789760692.008397:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681286973901 time 681286713485 state 0x1 ref 1 00000100:00100000:0.0:1789760692.008438:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681286973901 time 681339368104 state 0x0 ref 1 00000100:00100000:0.0:1789760692.008463:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484110848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760692.008537:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17657856, mtime: 0x6aad94b3, atime: 0x0, ctime: 0x6aad94b3, blocks: 0x8807 00010000:00010000:0.0:1789760692.008549:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f9e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.008570:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=681s v=(0 0) 00010000:00010000:0.0:1789760692.008576:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760692.008584:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f9e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17657856->17661951] (req 17657856->17661951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.008608:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396206f9e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17657856->17661951] (req 17657856->17661951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760692.111749:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1789760692.111788:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f47480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508898816:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760692.111884:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760692.112440:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1789760692.112452:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508898944:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760692.112501:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760692.112539:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508898944 00000100:00100000:0.0:1789760692.112547:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508898944 00000100:00100000:0.0:1789760692.112610:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 401 00000100:00100000:0.0:1789760692.112622:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508898944:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:0.0:1789760692.112668:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:0.0:1789760692.112732:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508898944:12345-0@lo:41:osp-pre-0-1.0 Request processed in 112us (229us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760692.112739:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 401 00000100:00100000:0.0:1789760692.112797:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508898944:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760692.113187:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508899072:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760692.113222:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760692.113265:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508898816 00000100:00100000:0.0:1789760692.113270:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508898816 00000100:00100000:0.0:1789760692.113310:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508899072 00000100:00100000:0.0:1789760692.113313:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508899072 00000100:00100000:0.0:1789760692.113332:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 342 00000100:00100000:0.0:1789760692.113339:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508898816:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:0.0:1789760692.113358:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528023552 pending 1703936 free 3840933888 avail 3833593856 00000100:00100000:0.0:1789760692.113395:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508898816:12345-0@lo:13:osp-pre-0-0.0 Request processed in 56us (1511us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760692.113402:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 342 00000100:00100000:0.0:1789760692.113415:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 343 00000100:00100000:0.0:1789760692.113422:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b18e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508899072:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:0.0:1789760692.113432:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528023552 pending 1703936 free 3840933888 avail 3833593856 00000100:00100000:0.0:1789760692.113458:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508899072:12345-0@lo:13:osp-pre-0-1.0 Request processed in 37us (236us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760692.113478:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 343 00000100:00100000:0.0:1789760692.113512:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508899072:0@lo:13:osp-pre-0-1.0 00000100:00080000:0.0:1789760692.113528:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760692.113549:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f47480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508898816:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760692.191966:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484111104 00000100:00100000:0.0:1789760692.191977:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484111104 00000100:00100000:0.0:1789760692.192036:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681339628520 time 681435294054 state 0x0 ref 1 00000100:00100000:0.0:1789760692.192062:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681435554470 time 681435294054 state 0x1 ref 1 00000100:00100000:0.0:1789760692.192120:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681435554470 time 681523051193 state 0x0 ref 1 00000100:00100000:0.0:1789760692.192141:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12af800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484111104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760692.192190:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17666048, mtime: 0x6aad94b3, atime: 0x0, ctime: 0x6aad94b3, blocks: 0x8807 00010000:00010000:0.0:1789760692.192199:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206fac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.192216:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=681s v=(0 0) 00010000:00010000:0.0:1789760692.192220:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760692.192227:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206fac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17666048->17670143] (req 17666048->17670143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.192383:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396206fac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17666048->17670143] (req 17666048->17670143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760692.230041:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760692.333978:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484111488 00000100:00100000:0.0:1789760692.333988:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484111488 00000100:00100000:0.0:1789760692.334046:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681561540668 time 681607319406 state 0x0 ref 1 00000100:00100000:0.0:1789760692.334061:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681607579822 time 681607319406 state 0x1 ref 1 00000100:00100000:0.0:1789760692.334117:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681607579822 time 681665047682 state 0x0 ref 1 00000100:00100000:0.0:1789760692.334132:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484111488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760692.334182:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17678336, mtime: 0x6aad94b4, atime: 0x0, ctime: 0x6aad94b4, blocks: 0x8807 00010000:00010000:0.0:1789760692.334191:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.334207:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=681s v=(0 0) 00010000:00010000:0.0:1789760692.334212:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760692.334218:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17678336->17682431] (req 17678336->17682431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.334232:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17678336->17682431] (req 17678336->17682431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760692.387974:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17678336 -> 17682432 00010000:00010000:0.0:1789760692.387990:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17678336->17682431] (req 17678336->17682431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.388003:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17678336->17682431] (req 17678336->17682431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.388012:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17678336->17682431] (req 17678336->17682431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.388020:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760692.388172:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17678336->17682431] (req 17678336->17682431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.388183:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17678336->17682431] (req 17678336->17682431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.388204:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17678336->17682431] (req 17678336->17682431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.388212:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fc1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17678336->17682431] (req 17678336->17682431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760692.394269:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484111488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60128us (60583us total) trans 4294971634 rc 0/0 00000100:00100000:0.0:1789760692.394283:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4345 00000100:00100000:0.0:1789760692.394294:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681607579822 time 681665047682 state 0x2 ref 0 00000100:00100000:0.0:1789760692.409891:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484111616 00000100:00100000:0.0:1789760692.409908:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484111616 00000001:00080000:0.0:1789760692.448707:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971635 is committed 00000001:00080000:0.0:1789760692.563736:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971637 is committed 00000100:00100000:0.0:1789760692.576470:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484112000 00000100:00100000:0.0:1789760692.576479:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484112000 00000100:00100000:0.0:1789760692.576537:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681789987370 time 681860902827 state 0x0 ref 1 00000100:00100000:0.0:1789760692.576553:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681861163243 time 681860902827 state 0x1 ref 1 00000100:00100000:0.0:1789760692.576589:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681861163243 time 681907519763 state 0x0 ref 1 00000100:00100000:0.0:1789760692.576603:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484112000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760692.576652:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17694720, mtime: 0x6aad94b4, atime: 0x0, ctime: 0x6aad94b4, blocks: 0x8807 00010000:00010000:0.0:1789760692.576661:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.576676:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=681s v=(0 0) 00010000:00010000:0.0:1789760692.576690:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760692.576696:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17694720->17698815] (req 17694720->17698815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.576715:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17694720->17698815] (req 17694720->17698815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760692.653661:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17694720 -> 17698816 00010000:00010000:0.0:1789760692.653692:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17694720->17698815] (req 17694720->17698815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.653705:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17694720->17698815] (req 17694720->17698815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.653715:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17694720->17698815] (req 17694720->17698815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.653722:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760692.653727:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17694720->17698815] (req 17694720->17698815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.653736:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17694720->17698815] (req 17694720->17698815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.653757:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17694720->17698815] (req 17694720->17698815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.653766:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206fdd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17694720->17698815] (req 17694720->17698815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760692.653946:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484112000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77340us (77729us total) trans 4294971638 rc 0/0 00000100:00100000:0.0:1789760692.653957:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4349 00000100:00100000:0.0:1789760692.653967:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681861163243 time 681907519763 state 0x2 ref 0 00000100:00100000:0.0:1789760692.881971:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484112256 00000100:00100000:0.0:1789760692.881985:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484112256 00000100:00100000:0.0:1789760692.882060:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 681907780179 time 682009213796 state 0x0 ref 1 00000100:00100000:0.0:1789760692.882129:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682009474212 time 682009213796 state 0x1 ref 1 00000100:00100000:0.0:1789760692.882157:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682009474212 time 682213087654 state 0x0 ref 1 00000100:00100000:0.0:1789760692.882173:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484112256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760692.882229:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17702912, mtime: 0x6aad94b4, atime: 0x0, ctime: 0x6aad94b4, blocks: 0x8807 00010000:00010000:0.0:1789760692.882237:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206feb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.882254:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=682s v=(0 0) 00010000:00010000:0.0:1789760692.882258:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760692.882265:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206feb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17702912->17707007] (req 17702912->17707007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.882279:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396206feb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17702912->17707007] (req 17702912->17707007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760692.968571:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484112384 00000100:00100000:0.0:1789760692.968581:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484112384 00000100:00100000:0.0:1789760692.968641:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682009474212 time 682213087654 state 0x0 ref 1 00000100:00100000:0.0:1789760692.968656:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682213348070 time 682213087654 state 0x1 ref 1 00000100:00100000:0.0:1789760692.968692:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682213348070 time 682299622469 state 0x0 ref 1 00000100:00100000:0.0:1789760692.968706:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ac380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484112384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760692.968754:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17707008, mtime: 0x6aad94b4, atime: 0x0, ctime: 0x6aad94b4, blocks: 0x8807 00010000:00010000:0.0:1789760692.968763:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.968779:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=682s v=(0 0) 00010000:00010000:0.0:1789760692.968784:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760692.968789:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17707008->17711103] (req 17707008->17711103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760692.968802:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17707008->17711103] (req 17707008->17711103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760693.055053:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17707008 -> 17711104 00010000:00010000:0.0:1789760693.055086:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17707008->17711103] (req 17707008->17711103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.055106:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17707008->17711103] (req 17707008->17711103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.055116:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17707008->17711103] (req 17707008->17711103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.055123:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760693.055127:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17707008->17711103] (req 17707008->17711103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.055136:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17707008->17711103] (req 17707008->17711103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.055154:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17707008->17711103] (req 17707008->17711103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.055163:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17707008->17711103] (req 17707008->17711103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760693.055339:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ac380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484112384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 86632us (86932us total) trans 4294971641 rc 0/0 00000100:00100000:0.0:1789760693.055350:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4352 00000100:00100000:0.0:1789760693.055367:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682213348070 time 682299622469 state 0x2 ref 0 00000100:00100000:0.0:1789760693.063738:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484112512 00000100:00100000:0.0:1789760693.063746:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484112512 00000100:00100000:0.0:1789760693.063802:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682213348070 time 682299622469 state 0x0 ref 1 00000100:00100000:0.0:1789760693.063816:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682299882885 time 682299622469 state 0x1 ref 1 00000100:00100000:0.0:1789760693.063838:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682299882885 time 682394769901 state 0x0 ref 1 00000100:00100000:0.0:1789760693.063850:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484112512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760693.063893:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17711104, mtime: 0x6aad94b4, atime: 0x0, ctime: 0x6aad94b4, blocks: 0x8807 00010000:00010000:0.0:1789760693.063901:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.063916:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=682s v=(0 0) 00010000:00010000:0.0:1789760693.063925:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760693.063931:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17711104->17715199] (req 17711104->17715199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.063946:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396206ff9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17711104->17715199] (req 17711104->17715199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.133922:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760693.151564:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484112640 00000100:00100000:0.0:1789760693.151576:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484112640 00000100:00100000:0.0:1789760693.151642:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682299882885 time 682394769901 state 0x0 ref 1 00000100:00100000:0.0:1789760693.151657:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682395030317 time 682394769901 state 0x1 ref 1 00000100:00100000:0.0:1789760693.151701:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682395030317 time 682482632426 state 0x0 ref 1 00000100:00100000:0.0:1789760693.151715:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484112640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760693.151760:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17715200, mtime: 0x6aad94b4, atime: 0x0, ctime: 0x6aad94b4, blocks: 0x8807 00010000:00010000:0.0:1789760693.151767:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207000 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.151782:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=682s v=(0 0) 00010000:00010000:0.0:1789760693.151786:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760693.151791:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207000 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17715200->17719295] (req 17715200->17719295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.151804:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207000 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17715200->17719295] (req 17715200->17719295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760693.501725:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484113152 00000100:00100000:0.0:1789760693.501739:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484113152 00000100:00100000:0.0:1789760693.501908:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682708191448 time 682782104038 state 0x0 ref 1 00000100:00100000:0.0:1789760693.501928:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682782364454 time 682782104038 state 0x1 ref 1 00000100:00100000:0.0:1789760693.501966:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682782364454 time 682832897599 state 0x0 ref 1 00000100:00100000:0.0:1789760693.501983:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484113152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760693.502054:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17731584, mtime: 0x6aad94b5, atime: 0x0, ctime: 0x6aad94b5, blocks: 0x8807 00010000:00010000:0.0:1789760693.502275:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620701c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.502303:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=682s v=(0 0) 00010000:00010000:0.0:1789760693.502309:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760693.502316:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620701c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17731584->17735679] (req 17731584->17735679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.502403:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620701c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17731584->17735679] (req 17731584->17735679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760693.563200:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484113280 00000100:00100000:0.0:1789760693.563213:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484113280 00000100:00100000:0.0:1789760693.563299:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682782364454 time 682832897599 state 0x0 ref 1 00000100:00100000:0.0:1789760693.563317:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682833158015 time 682832897599 state 0x1 ref 1 00000100:00100000:0.0:1789760693.563366:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682833158015 time 682894295794 state 0x0 ref 1 00000100:00100000:0.0:1789760693.563382:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484113280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760693.563450:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17735680, mtime: 0x6aad94b5, atime: 0x0, ctime: 0x6aad94b5, blocks: 0x8807 00010000:00010000:0.0:1789760693.563471:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207023 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.563492:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=682s v=(0 0) 00010000:00010000:0.0:1789760693.563497:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760693.563506:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207023 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17735680->17739775] (req 17735680->17739775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.563531:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207023 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17735680->17739775] (req 17735680->17739775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760693.701946:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971649 is committed 00000100:00100000:0.0:1789760693.706985:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484113536 00000100:00100000:0.0:1789760693.706999:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484113536 00000100:00100000:0.0:1789760693.707111:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682894556210 time 682940929506 state 0x0 ref 1 00000100:00100000:0.0:1789760693.707131:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682941189922 time 682940929506 state 0x1 ref 1 00000100:00100000:0.0:1789760693.707163:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 682941189922 time 683038094370 state 0x0 ref 1 00000100:00100000:0.0:1789760693.707179:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484113536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760693.707242:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17743872, mtime: 0x6aad94b5, atime: 0x0, ctime: 0x6aad94b5, blocks: 0x8807 00010000:00010000:0.0:1789760693.707253:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207031 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.707275:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=683s v=(0 0) 00010000:00010000:0.0:1789760693.707280:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760693.707288:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207031 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17743872->17747967] (req 17743872->17747967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.707305:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207031 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17743872->17747967] (req 17743872->17747967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760693.767501:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971650 is committed 00000100:00100000:0.0:1789760693.772346:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:0.0:1789760693.814981:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17747968 -> 17752064 00010000:00010000:0.0:1789760693.815008:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207038 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17747968->17752063] (req 17747968->17752063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.815024:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207038 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17747968->17752063] (req 17747968->17752063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.815034:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207038 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17747968->17752063] (req 17747968->17752063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.815041:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760693.815047:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207038 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17747968->17752063] (req 17747968->17752063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.815058:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207038 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17747968->17752063] (req 17747968->17752063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.815109:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207038 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17747968->17752063] (req 17747968->17752063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760693.815116:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207038 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17747968->17752063] (req 17747968->17752063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760693.815247:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484113664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42417us (42909us total) trans 4294971651 rc 0/0 00000100:00100000:0.0:1789760693.815259:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4362 00000100:00100000:0.0:1789760693.815271:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683038354786 time 683103677991 state 0x2 ref 0 00010000:00010000:2.0:1789760693.821211:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620703f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17752064->17756159] (req 17752064->17756159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.821230:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620703f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17752064->17756159] (req 17752064->17756159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760693.866448:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971652 is committed 00000100:00100000:2.0:1789760693.870880:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760693.904326:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971653 is committed 00002000:00010000:2.0:1789760693.904518:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17756160 -> 17760256 00010000:00010000:2.0:1789760693.904540:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207046 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17756160->17760255] (req 17756160->17760255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.904557:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207046 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17756160->17760255] (req 17756160->17760255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.904567:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207046 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17756160->17760255] (req 17756160->17760255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.904576:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760693.904581:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207046 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17756160->17760255] (req 17756160->17760255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.904592:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207046 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17756160->17760255] (req 17756160->17760255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.904618:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207046 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17756160->17760255] (req 17756160->17760255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.904626:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207046 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17756160->17760255] (req 17756160->17760255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760693.904754:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484113920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33496us (33882us total) trans 4294971653 rc 0/0 00000100:00100000:2.0:1789760693.904829:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4364 00000100:00100000:2.0:1789760693.904841:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683152236770 time 683202170297 state 0x2 ref 0 00000100:00100000:2.0:1789760693.909427:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484114048 00000100:00100000:2.0:1789760693.909441:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484114048 00000100:00100000:2.0:1789760693.909526:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683152236770 time 683202170297 state 0x0 ref 1 00000100:00100000:2.0:1789760693.909546:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683202430713 time 683202170297 state 0x1 ref 1 00000100:00100000:2.0:1789760693.909579:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683202430713 time 683240509959 state 0x0 ref 1 00000100:00100000:2.0:1789760693.909594:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484114048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760693.909660:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17760256, mtime: 0x6aad94b5, atime: 0x0, ctime: 0x6aad94b5, blocks: 0x8807 00010000:00010000:2.0:1789760693.909680:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.909701:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=683s v=(0 0) 00010000:00010000:2.0:1789760693.909705:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760693.909713:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17760256->17764351] (req 17760256->17764351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.909729:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17760256->17764351] (req 17760256->17764351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760693.946515:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971654 is committed 00002000:00010000:2.0:1789760693.946655:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17760256 -> 17764352 00010000:00010000:2.0:1789760693.946684:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17760256->17764351] (req 17760256->17764351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.946699:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17760256->17764351] (req 17760256->17764351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.946708:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17760256->17764351] (req 17760256->17764351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.946715:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760693.946722:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17760256->17764351] (req 17760256->17764351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.946732:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17760256->17764351] (req 17760256->17764351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.946826:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17760256->17764351] (req 17760256->17764351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760693.946836:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620704d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17760256->17764351] (req 17760256->17764351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760693.946976:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484114048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37377us (37642us total) trans 4294971654 rc 0/0 00000100:00100000:2.0:1789760693.946993:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4365 00000100:00100000:2.0:1789760693.947005:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683202430713 time 683240509959 state 0x2 ref 0 00000100:00100000:0.0:1789760693.950953:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760693.988714:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971655 is committed 00000100:00100000:0.0:1789760693.990649:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:2.0:1789760694.024274:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17768448 -> 17772544 00010000:00010000:2.0:1789760694.024303:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620705b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17768448->17772543] (req 17768448->17772543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.024318:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620705b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17768448->17772543] (req 17768448->17772543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.024328:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620705b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17768448->17772543] (req 17768448->17772543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.024335:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760694.024341:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620705b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17768448->17772543] (req 17768448->17772543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.024352:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620705b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17768448->17772543] (req 17768448->17772543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.024377:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620705b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17768448->17772543] (req 17768448->17772543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.024385:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620705b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17768448->17772543] (req 17768448->17772543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760694.024521:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484114304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33500us (33876us total) trans 4294971656 rc 0/0 00000100:00100000:2.0:1789760694.024534:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4367 00000100:00100000:2.0:1789760694.024547:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683282573715 time 683321931295 state 0x2 ref 0 00000001:00080000:2.0:1789760694.024596:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971656 is committed 00000100:00100000:2.0:1789760694.028593:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00080000:0.0:1789760694.032312:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760694.060974:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971657 is committed 00002000:00010000:0.0:1789760694.061138:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17772544 -> 17776640 00010000:00010000:0.0:1789760694.061159:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207062 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17772544->17776639] (req 17772544->17776639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.061174:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207062 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17772544->17776639] (req 17772544->17776639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.061185:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207062 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17772544->17776639] (req 17772544->17776639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.061192:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760694.061198:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207062 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17772544->17776639] (req 17772544->17776639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.061208:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207062 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17772544->17776639] (req 17772544->17776639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.061232:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207062 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17772544->17776639] (req 17772544->17776639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.061240:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207062 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17772544->17776639] (req 17772544->17776639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.061355:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484114432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32356us (32768us total) trans 4294971657 rc 0/0 00000100:00100000:0.0:1789760694.061368:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4368 00000100:00100000:0.0:1789760694.061381:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683322191711 time 683359909008 state 0x2 ref 0 00000100:00100000:0.0:1789760694.064256:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760694.064349:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484114560 00000100:00100000:2.0:1789760694.064362:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484114560 00000100:00100000:2.0:1789760694.064435:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683322191711 time 683359909008 state 0x0 ref 1 00000100:00100000:2.0:1789760694.064464:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683360169424 time 683359909008 state 0x1 ref 1 00000100:00100000:2.0:1789760694.064503:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683360169424 time 683395433819 state 0x0 ref 1 00000100:00100000:2.0:1789760694.064517:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484114560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760694.064582:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17776640, mtime: 0x6aad94b5, atime: 0x0, ctime: 0x6aad94b5, blocks: 0x8807 00010000:00010000:2.0:1789760694.064592:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.064611:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=683s v=(0 0) 00010000:00010000:2.0:1789760694.064616:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760694.064622:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17776640->17780735] (req 17776640->17780735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.064643:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17776640->17780735] (req 17776640->17780735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760694.092286:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17776640 -> 17780736 00010000:00010000:0.0:1789760694.092304:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17776640->17780735] (req 17776640->17780735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.092320:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17776640->17780735] (req 17776640->17780735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.092330:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17776640->17780735] (req 17776640->17780735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.092337:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760694.092343:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17776640->17780735] (req 17776640->17780735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.092358:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17776640->17780735] (req 17776640->17780735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.092384:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17776640->17780735] (req 17776640->17780735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.092394:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207069 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17776640->17780735] (req 17776640->17780735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.092607:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484114560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28085us (28359us total) trans 4294971658 rc 0/0 00000100:00100000:0.0:1789760694.092618:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4369 00000100:00100000:0.0:1789760694.092630:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683360169424 time 683395433819 state 0x2 ref 0 00000100:00100000:2.0:1789760694.096378:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:2.0:1789760694.123803:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17780736 -> 17784832 00010000:00010000:2.0:1789760694.123832:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207070 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17780736->17784831] (req 17780736->17784831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.123848:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207070 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17780736->17784831] (req 17780736->17784831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.123859:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207070 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17780736->17784831] (req 17780736->17784831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.123868:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760694.123874:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207070 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17780736->17784831] (req 17780736->17784831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.123885:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207070 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17780736->17784831] (req 17780736->17784831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.123911:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207070 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17780736->17784831] (req 17780736->17784831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.123919:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207070 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17780736->17784831] (req 17780736->17784831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760694.124047:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484114688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27310us (27678us total) trans 4294971659 rc 0/0 00000100:00100000:2.0:1789760694.124061:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4370 00000100:00100000:2.0:1789760694.124110:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683395694235 time 683427648844 state 0x2 ref 0 00000001:00080000:0.0:1789760694.126503:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971659 is committed 00000100:00100000:2.0:1789760694.126654:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00010000:00010000:0.0:1789760694.156740:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760694.166190:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484114944 00000100:00100000:0.0:1789760694.166202:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484114944 00000100:00100000:0.0:1789760694.166275:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683427909260 time 683457888960 state 0x0 ref 1 00000100:00100000:0.0:1789760694.166292:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683458149376 time 683457888960 state 0x1 ref 1 00000100:00100000:0.0:1789760694.166351:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683458149376 time 683497282091 state 0x0 ref 1 00000100:00100000:0.0:1789760694.166366:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484114944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760694.166429:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17788928, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x8807 00010000:00010000:0.0:1789760694.166440:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620707e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.166469:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=683s v=(0 0) 00010000:00010000:0.0:1789760694.166475:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760694.166483:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620707e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17788928->17793023] (req 17788928->17793023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.166500:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620707e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17788928->17793023] (req 17788928->17793023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.194841:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760694.194991:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484115072 00000100:00100000:2.0:1789760694.195002:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484115072 00000100:00100000:2.0:1789760694.195248:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683458149376 time 683497282091 state 0x0 ref 1 00000100:00100000:2.0:1789760694.195266:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683497542507 time 683497282091 state 0x1 ref 1 00000100:00100000:2.0:1789760694.195311:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683497542507 time 683526242352 state 0x0 ref 1 00000100:00100000:2.0:1789760694.195326:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484115072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760694.195394:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17793024, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x8807 00010000:00010000:2.0:1789760694.195404:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207085 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.195424:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=683s v=(0 0) 00010000:00010000:2.0:1789760694.195429:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760694.195435:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207085 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17793024->17797119] (req 17793024->17797119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.195451:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207085 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17793024->17797119] (req 17793024->17797119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760694.239896:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971662 is committed 00000100:00100000:1.0:1789760694.240336:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683497542507 time 683526242352 state 0x2 ref 0 00000100:00100000:1.0:1789760694.244662:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760694.296958:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971663 is committed 00002000:00010000:1.0:1789760694.297148:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17797120 -> 17801216 00010000:00010000:1.0:1789760694.297169:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620708c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17797120->17801215] (req 17797120->17801215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.297185:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620708c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17797120->17801215] (req 17797120->17801215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.297195:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620708c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17797120->17801215] (req 17797120->17801215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.297204:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760694.297211:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620708c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17797120->17801215] (req 17797120->17801215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.297221:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620708c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17797120->17801215] (req 17797120->17801215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.297249:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620708c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17797120->17801215] (req 17797120->17801215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.297257:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620708c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17797120->17801215] (req 17797120->17801215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760694.297397:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484115200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52414us (52736us total) trans 4294971663 rc 0/0 00000100:00100000:1.0:1789760694.297415:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4374 00000100:00100000:1.0:1789760694.297427:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683526502768 time 683575887466 state 0x2 ref 0 00000100:00100000:0.0:1789760694.300821:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760694.335421:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971664 is committed 00000100:00100000:0.0:1789760694.341813:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760694.367996:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971665 is committed 00002000:00010000:0.0:1789760694.368093:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17805312 -> 17809408 00010000:00010000:0.0:1789760694.368114:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620709a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17805312->17809407] (req 17805312->17809407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.368128:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620709a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17805312->17809407] (req 17805312->17809407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.368138:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620709a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17805312->17809407] (req 17805312->17809407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.368152:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760694.368157:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620709a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17805312->17809407] (req 17805312->17809407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.368167:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620709a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17805312->17809407] (req 17805312->17809407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.368192:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620709a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17805312->17809407] (req 17805312->17809407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.368200:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620709a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17805312->17809407] (req 17805312->17809407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.368311:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484115456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26055us (26553us total) trans 4294971665 rc 0/0 00000100:00100000:0.0:1789760694.368321:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4376 00000100:00100000:0.0:1789760694.368334:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683632290820 time 683672994234 state 0x2 ref 0 00000100:00100000:1.0:1789760694.370940:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760694.371041:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484115584 00000100:00100000:2.0:1789760694.371055:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484115584 00000100:00100000:2.0:1789760694.371180:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683632290820 time 683672994234 state 0x0 ref 1 00000100:00100000:2.0:1789760694.371200:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683673254650 time 683672994234 state 0x1 ref 1 00000100:00100000:2.0:1789760694.371234:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683673254650 time 683702164183 state 0x0 ref 1 00000100:00100000:2.0:1789760694.371249:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484115584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760694.371318:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17809408, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x8807 00010000:00010000:2.0:1789760694.371329:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.371350:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=683s v=(0 0) 00010000:00010000:2.0:1789760694.371355:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760694.371364:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17809408->17813503] (req 17809408->17813503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.371380:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17809408->17813503] (req 17809408->17813503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760694.405063:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17809408 -> 17813504 00010000:00010000:0.0:1789760694.405223:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17809408->17813503] (req 17809408->17813503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.405238:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17809408->17813503] (req 17809408->17813503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.405253:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17809408->17813503] (req 17809408->17813503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.405261:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760694.405267:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17809408->17813503] (req 17809408->17813503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.405276:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17809408->17813503] (req 17809408->17813503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.405300:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17809408->17813503] (req 17809408->17813503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.405310:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962070a1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17809408->17813503] (req 17809408->17813503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.406814:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484115584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35552us (35871us total) trans 4294971666 rc 0/0 00000100:00100000:0.0:1789760694.406831:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4377 00000100:00100000:0.0:1789760694.406844:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683673254650 time 683702164183 state 0x2 ref 0 00000100:00100000:1.0:1789760694.409775:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760694.441305:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971667 is committed 00002000:00010000:0.0:1789760694.441455:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17813504 -> 17817600 00010000:00010000:0.0:1789760694.441471:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962070a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17813504->17817599] (req 17813504->17817599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.441488:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962070a8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17813504->17817599] (req 17813504->17817599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.441495:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962070a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17813504->17817599] (req 17813504->17817599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.441500:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760694.441504:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962070a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17813504->17817599] (req 17813504->17817599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.441510:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962070a8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17813504->17817599] (req 17813504->17817599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.441529:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962070a8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17813504->17817599] (req 17813504->17817599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.441535:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962070a8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17813504->17817599] (req 17813504->17817599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.441646:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484115712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31455us (31883us total) trans 4294971667 rc 0/0 00000100:00100000:0.0:1789760694.441656:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4378 00000100:00100000:0.0:1789760694.441683:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683702424599 time 683741099197 state 0x2 ref 0 00000100:00100000:2.0:1789760694.446240:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760694.459514:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683702424599 time 683741099197 state 0x0 ref 1 00000100:00100000:2.0:1789760694.459544:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683741359613 time 683741099197 state 0x1 ref 1 00000100:00100000:2.0:1789760694.459585:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683741359613 time 683790515613 state 0x0 ref 1 00000100:00100000:2.0:1789760694.459602:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484115840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760694.459683:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17817600, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x8807 00010000:00010000:2.0:1789760694.459694:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.459713:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=683s v=(0 0) 00010000:00010000:2.0:1789760694.459718:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760694.461038:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17817600->17821695] (req 17817600->17821695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760694.461086:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17817600->17821695] (req 17817600->17821695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760694.545175:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971668 is committed 00002000:00010000:0.0:1789760694.545676:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17817600 -> 17821696 00010000:00010000:0.0:1789760694.545699:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17817600->17821695] (req 17817600->17821695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.545714:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17817600->17821695] (req 17817600->17821695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.545736:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17817600->17821695] (req 17817600->17821695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.545746:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760694.545752:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17817600->17821695] (req 17817600->17821695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.545762:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17817600->17821695] (req 17817600->17821695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.545788:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17817600->17821695] (req 17817600->17821695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.545797:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962070af lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17817600->17821695] (req 17817600->17821695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.545934:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484115840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 86330us (99702us total) trans 4294971668 rc 0/0 00000100:00100000:0.0:1789760694.545946:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4379 00000100:00100000:0.0:1789760694.545958:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683741359613 time 683790515613 state 0x2 ref 0 00000100:00100000:1.0:1789760694.551494:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760694.551992:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484115968 00000100:00100000:0.0:1789760694.552006:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484115968 00000100:00100000:0.0:1789760694.552123:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683741359613 time 683790515613 state 0x0 ref 1 00000100:00100000:0.0:1789760694.552142:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683790776029 time 683790515613 state 0x1 ref 1 00000100:00100000:0.0:1789760694.552183:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683790776029 time 683883112981 state 0x0 ref 1 00000100:00100000:0.0:1789760694.552198:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484115968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760694.552263:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17821696, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x8807 00010000:00010000:0.0:1789760694.552274:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.552296:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=683s v=(0 0) 00010000:00010000:0.0:1789760694.552301:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760694.552308:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17821696->17825791] (req 17821696->17825791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.552325:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17821696->17825791] (req 17821696->17825791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760694.590783:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17821696 -> 17825792 00010000:00010000:1.0:1789760694.590809:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17821696->17825791] (req 17821696->17825791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.590824:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17821696->17825791] (req 17821696->17825791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.590834:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17821696->17825791] (req 17821696->17825791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.590842:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760694.590847:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17821696->17825791] (req 17821696->17825791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.590857:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17821696->17825791] (req 17821696->17825791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.590882:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17821696->17825791] (req 17821696->17825791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.590890:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962070b6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17821696->17825791] (req 17821696->17825791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760694.591037:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484115968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38835us (39549us total) trans 4294971669 rc 0/0 00000100:00100000:1.0:1789760694.591050:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4380 00000100:00100000:1.0:1789760694.591105:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683790776029 time 683883112981 state 0x2 ref 0 00000001:00080000:1.0:1789760694.591166:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971669 is committed 00000100:00100000:1.0:1789760694.600131:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760694.600238:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484116096 00000100:00100000:0.0:1789760694.600250:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484116096 00000100:00100000:0.0:1789760694.600331:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683790776029 time 683883112981 state 0x0 ref 1 00000100:00100000:0.0:1789760694.600350:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683883373397 time 683883112981 state 0x1 ref 1 00000100:00100000:0.0:1789760694.600434:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683883373397 time 683931363841 state 0x0 ref 1 00000100:00100000:0.0:1789760694.600451:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484116096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760694.600530:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17825792, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x8807 00010000:00010000:0.0:1789760694.600542:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.600561:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=683s v=(0 0) 00010000:00010000:0.0:1789760694.600566:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760694.600574:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17825792->17829887] (req 17825792->17829887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.600591:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17825792->17829887] (req 17825792->17829887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760694.643227:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971670 is committed 00002000:00010000:1.0:1789760694.643277:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17825792 -> 17829888 00002000:00010000:1.0:1789760694.643284:0:8108:0:(ofd_lvb.c:331:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb blocks from disk: 34823 -> 36871 00010000:00010000:1.0:1789760694.643301:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17825792->17829887] (req 17825792->17829887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.643318:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17825792->17829887] (req 17825792->17829887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.643329:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17825792->17829887] (req 17825792->17829887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.643338:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760694.643344:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17825792->17829887] (req 17825792->17829887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.643362:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17825792->17829887] (req 17825792->17829887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.643388:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17825792->17829887] (req 17825792->17829887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.643402:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962070bd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17825792->17829887] (req 17825792->17829887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760694.643551:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484116096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43096us (43428us total) trans 4294971670 rc 0/0 00000100:00100000:1.0:1789760694.643568:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4381 00000100:00100000:1.0:1789760694.643582:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683883373397 time 683931363841 state 0x2 ref 0 00000100:00100000:2.0:1789760694.646195:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760694.670230:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971671 is committed 00000100:00100000:2.0:1789760694.673463:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760694.673588:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484116352 00000100:00100000:0.0:1789760694.673605:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484116352 00000100:00100000:0.0:1789760694.673730:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683931624257 time 683977518515 state 0x0 ref 1 00000100:00100000:0.0:1789760694.673870:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683977778931 time 683977518515 state 0x1 ref 1 00000100:00100000:0.0:1789760694.673912:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683977778931 time 684004843072 state 0x0 ref 1 00000100:00100000:0.0:1789760694.673927:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484116352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760694.673990:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17833984, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x9007 00010000:00010000:0.0:1789760694.674001:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.674022:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:0.0:1789760694.674027:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760694.674035:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17833984->17838079] (req 17833984->17838079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.674052:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17833984->17838079] (req 17833984->17838079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760694.702167:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971672 is committed 00002000:00010000:1.0:1789760694.702367:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17833984 -> 17838080 00010000:00010000:1.0:1789760694.702384:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17833984->17838079] (req 17833984->17838079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.702395:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17833984->17838079] (req 17833984->17838079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.702402:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17833984->17838079] (req 17833984->17838079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.702407:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760694.702412:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17833984->17838079] (req 17833984->17838079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.702420:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17833984->17838079] (req 17833984->17838079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.702437:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17833984->17838079] (req 17833984->17838079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.702442:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962070cb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17833984->17838079] (req 17833984->17838079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760694.702582:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484116352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28655us (29129us total) trans 4294971672 rc 0/0 00000100:00100000:1.0:1789760694.702591:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4383 00000100:00100000:1.0:1789760694.702600:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683977778931 time 684004843072 state 0x2 ref 0 00000100:00100000:1.0:1789760694.706338:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760694.706506:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484116480 00000100:00100000:0.0:1789760694.706518:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484116480 00000100:00100000:0.0:1789760694.706634:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 683977778931 time 684004843072 state 0x0 ref 1 00000100:00100000:0.0:1789760694.706648:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684005103488 time 684004843072 state 0x1 ref 1 00000100:00100000:0.0:1789760694.706727:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684005103488 time 684037656114 state 0x0 ref 1 00000100:00100000:0.0:1789760694.706744:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484116480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760694.706813:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17838080, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x9007 00010000:00010000:0.0:1789760694.706822:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.706838:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:0.0:1789760694.706842:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760694.706848:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17838080->17842175] (req 17838080->17842175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.706861:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17838080->17842175] (req 17838080->17842175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760694.734679:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971673 is committed 00002000:00010000:0.0:1789760694.734809:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17838080 -> 17842176 00010000:00010000:0.0:1789760694.734827:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17838080->17842175] (req 17838080->17842175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.734843:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17838080->17842175] (req 17838080->17842175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.734853:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17838080->17842175] (req 17838080->17842175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.734861:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760694.734866:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17838080->17842175] (req 17838080->17842175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.734877:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17838080->17842175] (req 17838080->17842175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.734901:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17838080->17842175] (req 17838080->17842175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.734915:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070d2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17838080->17842175] (req 17838080->17842175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.735038:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484116480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28292us (28709us total) trans 4294971673 rc 0/0 00000100:00100000:0.0:1789760694.735050:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4384 00000100:00100000:0.0:1789760694.735062:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684005103488 time 684037656114 state 0x2 ref 0 00000100:00100000:2.0:1789760694.739520:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760694.740231:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484116608 00000100:00100000:1.0:1789760694.740247:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484116608 00000100:00100000:1.0:1789760694.740323:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684005103488 time 684037656114 state 0x0 ref 1 00000100:00100000:1.0:1789760694.740344:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684037916530 time 684037656114 state 0x1 ref 1 00000100:00100000:1.0:1789760694.740386:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684037916530 time 684071316149 state 0x0 ref 1 00000100:00100000:1.0:1789760694.740400:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484116608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760694.740476:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17842176, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x9007 00010000:00010000:1.0:1789760694.740489:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.740510:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:1.0:1789760694.740515:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760694.740523:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17842176->17846271] (req 17842176->17846271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.740540:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17842176->17846271] (req 17842176->17846271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760694.774865:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17842176 -> 17846272 00010000:00010000:0.0:1789760694.774884:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17842176->17846271] (req 17842176->17846271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.774900:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17842176->17846271] (req 17842176->17846271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.774911:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17842176->17846271] (req 17842176->17846271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.774919:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760694.774924:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17842176->17846271] (req 17842176->17846271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.774935:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17842176->17846271] (req 17842176->17846271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.774960:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17842176->17846271] (req 17842176->17846271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.774974:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962070d9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17842176->17846271] (req 17842176->17846271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.777039:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484116608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36621us (37513us total) trans 4294971674 rc 0/0 00000100:00100000:0.0:1789760694.777058:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4385 00000100:00100000:0.0:1789760694.777145:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684037916530 time 684071316149 state 0x2 ref 0 00000100:00100000:2.0:1789760694.784705:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760694.836159:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971675 is committed 00000100:00100000:1.0:1789760694.839006:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760694.872548:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971676 is committed 00002000:00010000:1.0:1789760694.873175:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17850368 -> 17854464 00010000:00010000:1.0:1789760694.873197:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962070e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17850368->17854463] (req 17850368->17854463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.873211:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962070e7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17850368->17854463] (req 17850368->17854463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.873221:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962070e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17850368->17854463] (req 17850368->17854463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.873230:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760694.873236:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962070e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17850368->17854463] (req 17850368->17854463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.873252:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962070e7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17850368->17854463] (req 17850368->17854463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.873277:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962070e7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17850368->17854463] (req 17850368->17854463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.873285:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962070e7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17850368->17854463] (req 17850368->17854463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760694.873418:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484116864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34135us (34419us total) trans 4294971676 rc 0/0 00000100:00100000:1.0:1789760694.873431:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4387 00000100:00100000:1.0:1789760694.873444:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684116201102 time 684170193651 state 0x2 ref 0 00000100:00100000:2.0:1789760694.879022:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760694.879334:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484116992 00000100:00100000:0.0:1789760694.879348:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484116992 00000100:00100000:0.0:1789760694.879432:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684116201102 time 684170193651 state 0x0 ref 1 00000100:00100000:0.0:1789760694.879451:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684170454067 time 684170193651 state 0x1 ref 1 00000100:00100000:0.0:1789760694.879488:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684170454067 time 684210418327 state 0x0 ref 1 00000100:00100000:0.0:1789760694.879505:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484116992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760694.879578:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17854464, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x9007 00010000:00010000:0.0:1789760694.879589:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.879609:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:0.0:1789760694.879614:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760694.879622:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17854464->17858559] (req 17854464->17858559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.879646:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17854464->17858559] (req 17854464->17858559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760694.914476:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971677 is committed 00002000:00010000:1.0:1789760694.914609:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17854464 -> 17858560 00010000:00010000:1.0:1789760694.914627:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17854464->17858559] (req 17854464->17858559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.914642:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17854464->17858559] (req 17854464->17858559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.914653:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17854464->17858559] (req 17854464->17858559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.914661:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760694.914677:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17854464->17858559] (req 17854464->17858559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.914689:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17854464->17858559] (req 17854464->17858559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.914712:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17854464->17858559] (req 17854464->17858559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.914721:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962070ee lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17854464->17858559] (req 17854464->17858559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760694.914874:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484116992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35368us (35860us total) trans 4294971677 rc 0/0 00000100:00100000:1.0:1789760694.914887:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4388 00000100:00100000:1.0:1789760694.914899:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684170454067 time 684210418327 state 0x2 ref 0 00000100:00100000:2.0:1789760694.917931:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760694.918027:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484117120 00000100:00100000:0.0:1789760694.918040:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484117120 00000100:00100000:0.0:1789760694.918141:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684170454067 time 684210418327 state 0x0 ref 1 00000100:00100000:0.0:1789760694.918160:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684210678743 time 684210418327 state 0x1 ref 1 00000100:00100000:0.0:1789760694.918188:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684210678743 time 684249119252 state 0x0 ref 1 00000100:00100000:0.0:1789760694.918202:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484117120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760694.918262:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17858560, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x9007 00010000:00010000:0.0:1789760694.918273:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.918292:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:0.0:1789760694.918297:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760694.918304:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17858560->17862655] (req 17858560->17862655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.918321:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17858560->17862655] (req 17858560->17862655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760694.958955:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17858560 -> 17862656 00010000:00010000:0.0:1789760694.958979:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17858560->17862655] (req 17858560->17862655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.958993:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17858560->17862655] (req 17858560->17862655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.959003:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17858560->17862655] (req 17858560->17862655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.959012:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760694.959017:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17858560->17862655] (req 17858560->17862655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.959027:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17858560->17862655] (req 17858560->17862655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.959050:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17858560->17862655] (req 17858560->17862655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760694.959057:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962070f5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17858560->17862655] (req 17858560->17862655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760694.959205:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484117120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40998us (41279us total) trans 4294971678 rc 0/0 00000100:00100000:0.0:1789760694.959216:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4389 00000100:00100000:0.0:1789760694.959228:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684210678743 time 684249119252 state 0x2 ref 0 00000001:00080000:1.0:1789760694.959845:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971678 is committed 00000100:00100000:0.0:1789760694.962032:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760694.985230:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971679 is committed 00000100:00100000:1.0:1789760694.987674:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484117376 00000100:00100000:1.0:1789760694.987687:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484117376 00000100:00100000:1.0:1789760694.987756:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684249379668 time 684293245193 state 0x0 ref 1 00000100:00100000:1.0:1789760694.987775:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684293505609 time 684293245193 state 0x1 ref 1 00000100:00100000:1.0:1789760694.987810:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684293505609 time 684318740373 state 0x0 ref 1 00000100:00100000:1.0:1789760694.987823:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484117376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760694.987890:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17866752, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x9007 00010000:00010000:1.0:1789760694.987900:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.987919:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:1.0:1789760694.987925:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760694.987932:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17866752->17870847] (req 17866752->17870847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760694.987948:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17866752->17870847] (req 17866752->17870847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760695.037083:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971680 is committed 00002000:00010000:0.0:1789760695.037263:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17866752 -> 17870848 00010000:00010000:0.0:1789760695.037284:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17866752->17870847] (req 17866752->17870847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.037300:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17866752->17870847] (req 17866752->17870847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.037311:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17866752->17870847] (req 17866752->17870847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.037328:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760695.037334:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17866752->17870847] (req 17866752->17870847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.037346:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17866752->17870847] (req 17866752->17870847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.037375:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17866752->17870847] (req 17866752->17870847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.037385:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207103 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17866752->17870847] (req 17866752->17870847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760695.037507:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484117376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49679us (49996us total) trans 4294971680 rc 0/0 00000100:00100000:0.0:1789760695.037525:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4391 00000100:00100000:0.0:1789760695.037538:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684293505609 time 684318740373 state 0x2 ref 0 00000100:00100000:1.0:1789760695.045799:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760695.046023:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484117504 00000100:00100000:0.0:1789760695.046037:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484117504 00000100:00100000:0.0:1789760695.046269:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684293505609 time 684318740373 state 0x0 ref 1 00000100:00100000:0.0:1789760695.046283:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684319000789 time 684318740373 state 0x1 ref 1 00000100:00100000:0.0:1789760695.046312:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684319000789 time 684377243457 state 0x0 ref 1 00000100:00100000:0.0:1789760695.046331:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484117504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760695.046396:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17870848, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x9007 00010000:00010000:0.0:1789760695.046407:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.046426:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:0.0:1789760695.046432:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760695.046440:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17870848->17874943] (req 17870848->17874943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.046455:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17870848->17874943] (req 17870848->17874943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760695.075532:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17870848 -> 17874944 00010000:00010000:1.0:1789760695.075560:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17870848->17874943] (req 17870848->17874943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.075575:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17870848->17874943] (req 17870848->17874943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.075586:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17870848->17874943] (req 17870848->17874943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.075594:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760695.075611:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17870848->17874943] (req 17870848->17874943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.075627:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17870848->17874943] (req 17870848->17874943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.075652:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17870848->17874943] (req 17870848->17874943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.075671:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620710a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17870848->17874943] (req 17870848->17874943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760695.075804:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484117504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29469us (30012us total) trans 4294971681 rc 0/0 00000100:00100000:1.0:1789760695.075817:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4392 00000100:00100000:1.0:1789760695.075833:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684319000789 time 684377243457 state 0x2 ref 0 00000001:00080000:2.0:1789760695.075862:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971681 is committed 00000100:00100000:2.0:1789760695.080929:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760695.081646:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484117632 00000100:00100000:0.0:1789760695.081668:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484117632 00000100:00100000:0.0:1789760695.081809:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684319000789 time 684377243457 state 0x0 ref 1 00000100:00100000:0.0:1789760695.081830:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684377503873 time 684377243457 state 0x1 ref 1 00000100:00100000:0.0:1789760695.081919:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684377503873 time 684412848642 state 0x0 ref 1 00000100:00100000:0.0:1789760695.081936:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484117632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760695.082001:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17874944, mtime: 0x6aad94b6, atime: 0x0, ctime: 0x6aad94b6, blocks: 0x9007 00010000:00010000:0.0:1789760695.082011:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.082031:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:0.0:1789760695.082035:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760695.082043:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17874944->17879039] (req 17874944->17879039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.082059:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17874944->17879039] (req 17874944->17879039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760695.119466:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971682 is committed 00002000:00010000:0.0:1789760695.119674:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17874944 -> 17879040 00010000:00010000:0.0:1789760695.119695:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17874944->17879039] (req 17874944->17879039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.119710:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17874944->17879039] (req 17874944->17879039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.119720:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17874944->17879039] (req 17874944->17879039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.119730:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760695.119736:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17874944->17879039] (req 17874944->17879039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.119746:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17874944->17879039] (req 17874944->17879039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.119784:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17874944->17879039] (req 17874944->17879039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.119793:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207111 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17874944->17879039] (req 17874944->17879039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760695.119925:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484117632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37988us (39004us total) trans 4294971682 rc 0/0 00000100:00100000:0.0:1789760695.119938:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4393 00000100:00100000:0.0:1789760695.119951:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684377503873 time 684412848642 state 0x2 ref 0 00000100:00100000:0.0:1789760695.122618:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760695.166040:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971683 is committed 00002000:00010000:2.0:1789760695.166157:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17879040 -> 17883136 00002000:00010000:2.0:1789760695.166162:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760694 -> 1789760695 00002000:00010000:2.0:1789760695.166165:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760694 -> 1789760695 00010000:00010000:2.0:1789760695.166181:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207118 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17879040->17883135] (req 17879040->17883135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.166206:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207118 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17879040->17883135] (req 17879040->17883135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.166217:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207118 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17879040->17883135] (req 17879040->17883135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.166225:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760695.166231:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207118 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17879040->17883135] (req 17879040->17883135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.166241:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207118 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17879040->17883135] (req 17879040->17883135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.166267:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207118 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17879040->17883135] (req 17879040->17883135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.166280:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207118 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17879040->17883135] (req 17879040->17883135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760695.166400:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484117760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43467us (43790us total) trans 4294971683 rc 0/0 00000100:00100000:2.0:1789760695.166413:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4394 00000100:00100000:2.0:1789760695.166424:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684413109058 time 684453844810 state 0x2 ref 0 00000100:00100000:2.0:1789760695.169189:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00010000:00010000:1.0:1789760695.180332:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760695.213730:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971684 is committed 00002000:00010000:1.0:1789760695.213906:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17883136 -> 17887232 00010000:00010000:1.0:1789760695.213926:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620711f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17883136->17887231] (req 17883136->17887231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.213941:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620711f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17883136->17887231] (req 17883136->17887231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.213951:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620711f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17883136->17887231] (req 17883136->17887231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.213958:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760695.213965:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620711f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17883136->17887231] (req 17883136->17887231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.213975:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620711f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17883136->17887231] (req 17883136->17887231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.214000:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620711f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17883136->17887231] (req 17883136->17887231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.214008:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620711f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17883136->17887231] (req 17883136->17887231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760695.214199:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484117888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44764us (45018us total) trans 4294971684 rc 0/0 00000100:00100000:1.0:1789760695.214212:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4395 00000100:00100000:1.0:1789760695.214224:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684454105226 time 684500347073 state 0x2 ref 0 00000100:00100000:2.0:1789760695.217110:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760695.217259:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484118016 00000100:00100000:0.0:1789760695.217271:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484118016 00000100:00100000:0.0:1789760695.217349:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684454105226 time 684500347073 state 0x0 ref 1 00000100:00100000:0.0:1789760695.217368:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684500607489 time 684500347073 state 0x1 ref 1 00000100:00100000:0.0:1789760695.217542:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684500607489 time 684548375709 state 0x0 ref 1 00000100:00100000:0.0:1789760695.217560:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484118016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760695.217623:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17887232, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:0.0:1789760695.217633:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.217652:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:0.0:1789760695.217658:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760695.217676:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17887232->17891327] (req 17887232->17891327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.217692:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17887232->17891327] (req 17887232->17891327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760695.278525:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17887232 -> 17891328 00010000:00010000:0.0:1789760695.278550:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17887232->17891327] (req 17887232->17891327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.278565:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17887232->17891327] (req 17887232->17891327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.278575:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17887232->17891327] (req 17887232->17891327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.278583:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760695.278588:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17887232->17891327] (req 17887232->17891327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.278598:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17887232->17891327] (req 17887232->17891327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.278622:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17887232->17891327] (req 17887232->17891327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.278631:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207126 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17887232->17891327] (req 17887232->17891327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760695.278776:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484118016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61214us (61713us total) trans 4294971685 rc 0/0 00000100:00100000:0.0:1789760695.278788:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4396 00000100:00100000:0.0:1789760695.278800:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684500607489 time 684548375709 state 0x2 ref 0 00000001:00080000:1.0:1789760695.279398:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971685 is committed 00000100:00100000:0.0:1789760695.292221:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484118144 00000100:00100000:0.0:1789760695.292235:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484118144 00000100:00100000:0.0:1789760695.292315:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684500607489 time 684548375709 state 0x0 ref 1 00000100:00100000:0.0:1789760695.292334:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684548636125 time 684548375709 state 0x1 ref 1 00000100:00100000:0.0:1789760695.292362:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684548636125 time 684623291952 state 0x0 ref 1 00000100:00100000:0.0:1789760695.292377:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484118144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760695.292457:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17891328, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:0.0:1789760695.292469:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.292489:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:0.0:1789760695.292494:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760695.292501:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17891328->17895423] (req 17891328->17895423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.292518:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17891328->17895423] (req 17891328->17895423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760695.410162:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971686 is committed 00002000:00010000:2.0:1789760695.410286:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17891328 -> 17895424 00010000:00010000:2.0:1789760695.410306:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17891328->17895423] (req 17891328->17895423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.410321:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17891328->17895423] (req 17891328->17895423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.410332:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17891328->17895423] (req 17891328->17895423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.410341:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760695.410347:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17891328->17895423] (req 17891328->17895423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.410363:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17891328->17895423] (req 17891328->17895423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.410389:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17891328->17895423] (req 17891328->17895423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.410396:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620712d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17891328->17895423] (req 17891328->17895423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760695.410885:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484118144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 118503us (118854us total) trans 4294971686 rc 0/0 00000100:00100000:2.0:1789760695.410897:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4397 00000100:00100000:2.0:1789760695.410909:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684548636125 time 684623291952 state 0x2 ref 0 00000100:00100000:2.0:1789760695.424632:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760695.425764:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484118272 00000100:00100000:0.0:1789760695.425780:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484118272 00000100:00100000:0.0:1789760695.425870:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684548636125 time 684623291952 state 0x0 ref 1 00000100:00100000:0.0:1789760695.425889:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684623552368 time 684623291952 state 0x1 ref 1 00000100:00100000:0.0:1789760695.425929:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684623552368 time 684756860312 state 0x0 ref 1 00000100:00100000:0.0:1789760695.425947:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484118272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760695.426015:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17895424, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:0.0:1789760695.426026:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.426047:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:0.0:1789760695.426053:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760695.426061:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17895424->17899519] (req 17895424->17899519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.426134:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17895424->17899519] (req 17895424->17899519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760695.483264:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971687 is committed 00002000:00010000:0.0:1789760695.483369:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17895424 -> 17899520 00010000:00010000:0.0:1789760695.483389:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17895424->17899519] (req 17895424->17899519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.483405:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17895424->17899519] (req 17895424->17899519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.483417:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17895424->17899519] (req 17895424->17899519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.483427:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760695.483433:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17895424->17899519] (req 17895424->17899519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.483741:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17895424->17899519] (req 17895424->17899519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.483829:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17895424->17899519] (req 17895424->17899519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760695.483839:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207134 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17895424->17899519] (req 17895424->17899519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760695.484247:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484118272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58298us (59624us total) trans 4294971687 rc 0/0 00000100:00100000:0.0:1789760695.484260:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4398 00000100:00100000:0.0:1789760695.484273:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684623552368 time 684756860312 state 0x2 ref 0 00000100:00100000:1.0:1789760695.487605:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484118400 00000100:00100000:1.0:1789760695.487621:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484118400 00000100:00100000:1.0:1789760695.487816:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684623552368 time 684756860312 state 0x0 ref 1 00000100:00100000:1.0:1789760695.487835:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684757120728 time 684756860312 state 0x1 ref 1 00000100:00100000:1.0:1789760695.487915:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684757120728 time 684818845211 state 0x0 ref 1 00000100:00100000:1.0:1789760695.487930:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484118400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760695.487996:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17899520, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:1.0:1789760695.488017:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.488037:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:1.0:1789760695.488042:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760695.488050:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17899520->17903615] (req 17899520->17903615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.488093:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17899520->17903615] (req 17899520->17903615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760695.554188:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971688 is committed 00002000:00010000:2.0:1789760695.554331:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17899520 -> 17903616 00010000:00010000:2.0:1789760695.554352:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17899520->17903615] (req 17899520->17903615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.554369:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17899520->17903615] (req 17899520->17903615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.554380:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17899520->17903615] (req 17899520->17903615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.554393:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760695.554398:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17899520->17903615] (req 17899520->17903615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.554409:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17899520->17903615] (req 17899520->17903615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.554434:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17899520->17903615] (req 17899520->17903615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.554455:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620713b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17899520->17903615] (req 17899520->17903615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760695.554570:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484118400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66638us (67048us total) trans 4294971688 rc 0/0 00000100:00100000:2.0:1789760695.554582:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4399 00000100:00100000:2.0:1789760695.554595:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684757120728 time 684818845211 state 0x2 ref 0 00000100:00100000:0.0:1789760695.557451:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760695.557531:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484118528 00000100:00100000:2.0:1789760695.557544:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484118528 00000100:00100000:2.0:1789760695.557608:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684757120728 time 684818845211 state 0x0 ref 1 00000100:00100000:2.0:1789760695.557626:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684819105627 time 684818845211 state 0x1 ref 1 00000100:00100000:2.0:1789760695.557729:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684819105627 time 684888658680 state 0x0 ref 1 00000100:00100000:2.0:1789760695.557803:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484118528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760695.558047:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17903616, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:2.0:1789760695.558058:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.558103:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:2.0:1789760695.558108:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760695.558116:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17903616->17907711] (req 17903616->17907711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.558131:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17903616->17907711] (req 17903616->17907711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760695.605679:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17903616 -> 17907712 00010000:00010000:1.0:1789760695.605697:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17903616->17907711] (req 17903616->17907711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.605712:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17903616->17907711] (req 17903616->17907711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.605724:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17903616->17907711] (req 17903616->17907711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.605732:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760695.605738:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17903616->17907711] (req 17903616->17907711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.605849:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17903616->17907711] (req 17903616->17907711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.605877:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17903616->17907711] (req 17903616->17907711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.605887:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207142 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17903616->17907711] (req 17903616->17907711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760695.606858:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484118528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48259us (48575us total) trans 4294971689 rc 0/0 00000100:00100000:1.0:1789760695.606871:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4400 00000100:00100000:1.0:1789760695.606883:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684819105627 time 684888658680 state 0x2 ref 0 00000100:00100000:0.0:1789760695.610274:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:1.0:1789760695.659366:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17907712 -> 17911808 00010000:00010000:1.0:1789760695.659394:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207149 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17907712->17911807] (req 17907712->17911807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.659411:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207149 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17907712->17911807] (req 17907712->17911807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.659422:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207149 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17907712->17911807] (req 17907712->17911807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.659430:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760695.659435:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207149 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17907712->17911807] (req 17907712->17911807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.659462:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207149 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17907712->17911807] (req 17907712->17911807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.659487:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207149 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17907712->17911807] (req 17907712->17911807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.659496:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207149 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17907712->17911807] (req 17907712->17911807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760695.659627:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484118656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49089us (49358us total) trans 4294971690 rc 0/0 00000100:00100000:1.0:1789760695.659638:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4401 00000100:00100000:1.0:1789760695.659650:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684888919096 time 684941452214 state 0x2 ref 0 00000001:00080000:2.0:1789760695.662684:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971690 is committed 00000100:00100000:2.0:1789760695.663328:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760695.663408:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484118784 00000100:00100000:1.0:1789760695.663420:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484118784 00000100:00100000:1.0:1789760695.663493:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684888919096 time 684941452214 state 0x0 ref 1 00000100:00100000:1.0:1789760695.663512:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684941712630 time 684941452214 state 0x1 ref 1 00000100:00100000:1.0:1789760695.663543:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684941712630 time 684994473829 state 0x0 ref 1 00000100:00100000:1.0:1789760695.663555:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484118784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760695.663614:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17911808, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:1.0:1789760695.663623:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.663644:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=684s v=(0 0) 00010000:00010000:1.0:1789760695.663649:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760695.663672:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17911808->17915903] (req 17911808->17915903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.663689:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17911808->17915903] (req 17911808->17915903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760695.692406:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfbb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508899328:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760695.692509:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760695.692596:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508899328 00000100:00100000:0.0:1789760695.692604:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508899328 00000100:00100000:0.0:1789760695.692678:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 344 00000100:00100000:0.0:1789760695.692690:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fdb2f850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508899328:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:0.0:1789760695.692731:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760695.693222:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9470e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508899712:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.693262:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760695.693284:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8109f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0000_UUID:3479:1876699508900224:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.693306:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760695.693328:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c810a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0001_UUID:3479:1876699508900736:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.693348:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760695.693367:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:1.0:1789760695.693373:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9471f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:3479:1876699508899584:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.693394:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760695.693410:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8108a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:3479:1876699508900096:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.693428:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760695.693455:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8108000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:3479:1876699508900608:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.693474:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760695.693500:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508900224 00000100:00100000:1.0:1789760695.693505:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508900224 00000100:00100000:1.0:1789760695.693536:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508900736 00000100:00100000:1.0:1789760695.693539:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508900736 00000100:00100000:1.0:1789760695.693552:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508899584 00000100:00100000:1.0:1789760695.693554:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508899584 00000100:00100000:1.0:1789760695.693563:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508900096 00000100:00100000:1.0:1789760695.693565:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508900096 00000100:00100000:1.0:1789760695.693574:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508900608 00000100:00100000:1.0:1789760695.693576:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508900608 00000100:00100000:1.0:1789760695.693590:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 825 00000100:00100000:1.0:1789760695.693596:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0000_UUID+5:3479:x1876699508900224:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.693642:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0000_UUID+5:3479:x1876699508900224:12345-0@lo:400:kworker.0 Request processed in 46us (334us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760695.693649:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 825 00000100:00100000:1.0:1789760695.693683:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 826 00000100:00100000:1.0:1789760695.693689:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0001_UUID+5:3479:x1876699508900736:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.693713:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0001_UUID+5:3479:x1876699508900736:12345-0@lo:400:kworker.0 Request processed in 25us (365us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760695.693719:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 826 00000100:00100000:1.0:1789760695.693728:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 827 00000100:00100000:1.0:1789760695.693733:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3479:x1876699508899584:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760695.693900:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 828 00000100:00100000:2.0:1789760695.693912:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810aa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0000_UUID+5:3479:x1876699508900096:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760695.693954:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810aa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0000_UUID+5:3479:x1876699508900096:12345-0@lo:400:kworker.0 Request processed in 46us (524us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760695.693960:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 828 00000100:00100000:2.0:1789760695.693974:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 829 00000100:00100000:2.0:1789760695.693979:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0001_UUID+5:3479:x1876699508900608:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760695.694003:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0001_UUID+5:3479:x1876699508900608:12345-0@lo:400:kworker.0 Request processed in 24us (528us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760695.694009:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 829 00000100:00100000:0.0:1789760695.694169:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fdb2f850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508899328:12345-0@lo:13:osp-pre-1-0.0 Request processed in 1480us (1658us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760695.694179:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 344 00000100:00100000:1.0:1789760695.694902:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3479:x1876699508899584:12345-0@lo:400:kworker.0 Request processed in 1164us (1502us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760695.694912:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 827 00000100:00100000:1.0:1789760695.695057:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9471f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:3479:1876699508899584:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.695092:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8108a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:3479:1876699508900096:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.695108:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8108000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:3479:1876699508900608:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.695119:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8109f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0000_UUID:3479:1876699508900224:0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.695131:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c810a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0001_UUID:3479:1876699508900736:0@lo:400:kworker.0 00000100:00080000:0.0:1789760695.695302:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:0.0:1789760695.695309:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:0.0:1789760695.695313:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:0.0:1789760695.695339:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508899712 00000100:00100000:0.0:1789760695.695346:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508899712 00000100:00100000:0.0:1789760695.697464:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9471180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508899456:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.697545:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760695.697598:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c810ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508899968:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.697625:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760695.697644:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c810b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508900480:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.697686:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760695.697709:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8f5a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508900992:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760695.697739:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760695.697778:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [2->3] 00000100:00100000:0.0:1789760695.697785:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9470380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508899840:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.697809:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760695.699124:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c810b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508900352:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.699391:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760695.699436:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8109500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508900864:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.699460:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760695.699506:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9471180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508899456:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.699546:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508899968 00000100:00100000:0.0:1789760695.699552:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508899968 00000100:00100000:0.0:1789760695.699603:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 403 00000100:00100000:0.0:1789760695.699610:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508899968:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.699681:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508899968:12345-0@lo:400:kworker.0 Request processed in 70us (2054us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760695.699690:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 403 00000100:00100000:1.0:1789760695.700121:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508899840 00000100:00100000:1.0:1789760695.700127:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508899840 00000100:00100000:1.0:1789760695.700158:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 830 00000100:00100000:1.0:1789760695.700165:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb7b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508899840:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.700205:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb7b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508899840:12345-0@lo:400:kworker.0 Request processed in 42us (2396us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760695.700211:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 830 00000100:00100000:1.0:1789760695.700231:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508900992 00000100:00100000:1.0:1789760695.700235:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508900992 00000100:00100000:1.0:1789760695.700261:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 345 00000100:00100000:1.0:1789760695.700268:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508900992:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:1.0:1789760695.700284:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760695.700314:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508900992:12345-0@lo:13:osp-pre-1-1.0 Request processed in 47us (2577us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760695.700320:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 345 00000100:00100000:1.0:1789760695.700366:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfbb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508899328:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760695.700576:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 404 00000100:00100000:1.0:1789760695.700584:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108380 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508899712:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760695.700623:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108380 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508899712:12345-0@lo:400:kworker.0 Request processed in 41us (7361us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760695.700629:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 404 00000100:00100000:1.0:1789760695.700669:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9470e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508899712:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.701251:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508900864 00000100:00100000:0.0:1789760695.701257:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508900864 00000100:00100000:0.0:1789760695.701294:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 292 00000100:00100000:0.0:1789760695.701303:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508900480:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.701351:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508900480:12345-0@lo:400:kworker.0 Request processed in 50us (3666us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760695.701358:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 292 00000100:00100000:0.0:1789760695.701372:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 293 00000100:00100000:0.0:1789760695.701379:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508900352:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.701405:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508900352:12345-0@lo:400:kworker.0 Request processed in 27us (2018us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760695.701411:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 293 00000100:00100000:0.0:1789760695.701420:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 294 00000100:00100000:0.0:1789760695.701426:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508900864:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.701446:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508900864:12345-0@lo:400:kworker.0 Request processed in 21us (1986us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760695.701452:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 294 00000100:00080000:0.0:1789760695.701477:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:0.0:1789760695.701481:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760695.701513:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9470380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508899840:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.701525:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c810b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508900352:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.701536:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8109500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508900864:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.701547:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c810ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508899968:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.701559:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c810b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508900480:0@lo:400:kworker.0 00000100:00100000:0.0:1789760695.701574:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8f5a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508900992:0@lo:13:osp-pre-1-1.0 00000001:00080000:1.0:1789760695.703424:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971691 is committed 00002000:00010000:1.0:1789760695.703534:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17911808 -> 17915904 00010000:00010000:1.0:1789760695.703553:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17911808->17915903] (req 17911808->17915903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.703568:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17911808->17915903] (req 17911808->17915903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.703579:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17911808->17915903] (req 17911808->17915903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.703587:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760695.703592:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17911808->17915903] (req 17911808->17915903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.703601:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17911808->17915903] (req 17911808->17915903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.703628:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17911808->17915903] (req 17911808->17915903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.703636:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207150 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17911808->17915903] (req 17911808->17915903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760695.704052:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484118784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40384us (40616us total) trans 4294971691 rc 0/0 00000100:00100000:1.0:1789760695.704108:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4402 00000100:00100000:1.0:1789760695.704121:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684941712630 time 684994473829 state 0x2 ref 0 00000100:00100000:1.0:1789760695.708480:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760695.708550:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484118912 00000100:00100000:2.0:1789760695.708563:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484118912 00000100:00100000:2.0:1789760695.708626:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684941712630 time 684994473829 state 0x0 ref 1 00000100:00100000:2.0:1789760695.708644:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684994734245 time 684994473829 state 0x1 ref 1 00000100:00100000:2.0:1789760695.708682:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684994734245 time 685039612434 state 0x0 ref 1 00000100:00100000:2.0:1789760695.708695:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484118912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760695.708902:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17915904, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:2.0:1789760695.708914:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.708932:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:2.0:1789760695.708937:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760695.708952:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17915904->17919999] (req 17915904->17919999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.708968:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17915904->17919999] (req 17915904->17919999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760695.797867:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17915904 -> 17920000 00010000:00010000:2.0:1789760695.797889:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17915904->17919999] (req 17915904->17919999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.797906:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17915904->17919999] (req 17915904->17919999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.797922:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17915904->17919999] (req 17915904->17919999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.797932:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760695.797938:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17915904->17919999] (req 17915904->17919999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.797949:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17915904->17919999] (req 17915904->17919999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.797972:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17915904->17919999] (req 17915904->17919999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.797981:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207157 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17915904->17919999] (req 17915904->17919999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760695.798135:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484118912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 89435us (89660us total) trans 4294971692 rc 0/0 00000100:00100000:2.0:1789760695.798146:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4403 00000100:00100000:2.0:1789760695.798159:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684994734245 time 685039612434 state 0x2 ref 0 00000100:00100000:1.0:1789760695.803455:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760695.803537:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484119040 00000100:00100000:2.0:1789760695.803550:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484119040 00000100:00100000:2.0:1789760695.803617:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 684994734245 time 685039612434 state 0x0 ref 1 00000100:00100000:2.0:1789760695.803635:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685039872850 time 685039612434 state 0x1 ref 1 00000100:00100000:2.0:1789760695.803675:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685039872850 time 685134605712 state 0x0 ref 1 00000100:00100000:2.0:1789760695.803689:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484119040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760695.803751:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17920000, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:2.0:1789760695.803761:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.803780:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:2.0:1789760695.803786:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760695.803793:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17920000->17924095] (req 17920000->17924095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.803810:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17920000->17924095] (req 17920000->17924095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760695.857685:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971693 is committed 00002000:00010000:2.0:1789760695.858364:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17920000 -> 17924096 00010000:00010000:2.0:1789760695.858385:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17920000->17924095] (req 17920000->17924095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.858401:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17920000->17924095] (req 17920000->17924095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.858411:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17920000->17924095] (req 17920000->17924095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.858420:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760695.858427:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17920000->17924095] (req 17920000->17924095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.858447:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17920000->17924095] (req 17920000->17924095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.858472:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17920000->17924095] (req 17920000->17924095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.858480:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620715e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17920000->17924095] (req 17920000->17924095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760695.858608:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484119040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54914us (55160us total) trans 4294971693 rc 0/0 00000100:00100000:2.0:1789760695.858621:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4404 00000100:00100000:2.0:1789760695.858632:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685039872850 time 685134605712 state 0x2 ref 0 00000100:00100000:1.0:1789760695.872735:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484119168 00000100:00100000:1.0:1789760695.872756:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484119168 00000100:00100000:1.0:1789760695.872830:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685039872850 time 685134605712 state 0x0 ref 1 00000100:00100000:1.0:1789760695.872851:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685134866128 time 685134605712 state 0x1 ref 1 00000100:00100000:1.0:1789760695.872883:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685134866128 time 685203814082 state 0x0 ref 1 00000100:00100000:1.0:1789760695.872897:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484119168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760695.872969:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17924096, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:1.0:1789760695.872979:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.872998:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:1.0:1789760695.873003:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760695.873010:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17924096->17928191] (req 17924096->17928191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.873028:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17924096->17928191] (req 17924096->17928191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760695.946411:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17924096 -> 17928192 00010000:00010000:1.0:1789760695.946438:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17924096->17928191] (req 17924096->17928191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.946459:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17924096->17928191] (req 17924096->17928191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.946470:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17924096->17928191] (req 17924096->17928191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.946478:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760695.946484:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17924096->17928191] (req 17924096->17928191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.946494:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17924096->17928191] (req 17924096->17928191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.946519:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17924096->17928191] (req 17924096->17928191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.946529:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207165 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17924096->17928191] (req 17924096->17928191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760695.946567:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971694 is committed 00000100:00100000:1.0:1789760695.946719:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484119168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73817us (74213us total) trans 4294971694 rc 0/0 00000100:00100000:1.0:1789760695.946731:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4405 00000100:00100000:1.0:1789760695.946742:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685134866128 time 685203814082 state 0x2 ref 0 00000100:00100000:0.0:1789760695.951941:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760695.952121:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484119296 00000100:00100000:1.0:1789760695.952134:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484119296 00000100:00100000:1.0:1789760695.952201:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685134866128 time 685203814082 state 0x0 ref 1 00000100:00100000:1.0:1789760695.952218:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685204074498 time 685203814082 state 0x1 ref 1 00000100:00100000:1.0:1789760695.952252:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685204074498 time 685283182390 state 0x0 ref 1 00000100:00100000:1.0:1789760695.952265:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484119296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760695.952323:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17928192, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:1.0:1789760695.952334:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.952353:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:1.0:1789760695.952357:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760695.952366:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17928192->17932287] (req 17928192->17932287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.952382:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17928192->17932287] (req 17928192->17932287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760695.983528:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971695 is committed 00002000:00010000:1.0:1789760695.983700:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17928192 -> 17932288 00010000:00010000:1.0:1789760695.983718:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17928192->17932287] (req 17928192->17932287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.983734:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17928192->17932287] (req 17928192->17932287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.983810:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17928192->17932287] (req 17928192->17932287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.983819:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760695.983825:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17928192->17932287] (req 17928192->17932287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.983836:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17928192->17932287] (req 17928192->17932287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.983858:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17928192->17932287] (req 17928192->17932287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760695.983867:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620716c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17928192->17932287] (req 17928192->17932287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760695.984014:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484119296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31745us (32095us total) trans 4294971695 rc 0/0 00000100:00100000:1.0:1789760695.984026:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4406 00000100:00100000:1.0:1789760695.984037:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685204074498 time 685283182390 state 0x2 ref 0 00000100:00100000:0.0:1789760695.986675:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760695.986766:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484119424 00000100:00100000:2.0:1789760695.986778:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484119424 00000100:00100000:2.0:1789760695.986841:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685204074498 time 685283182390 state 0x0 ref 1 00000100:00100000:2.0:1789760695.986858:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685283442806 time 685283182390 state 0x1 ref 1 00000100:00100000:2.0:1789760695.986886:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685283442806 time 685317817304 state 0x0 ref 1 00000100:00100000:2.0:1789760695.986902:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484119424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760695.986957:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17932288, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:2.0:1789760695.986968:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.986986:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:2.0:1789760695.986991:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760695.986999:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17932288->17936383] (req 17932288->17936383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760695.987014:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17932288->17936383] (req 17932288->17936383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760696.028193:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17932288 -> 17936384 00010000:00010000:0.0:1789760696.028218:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17932288->17936383] (req 17932288->17936383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.028234:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17932288->17936383] (req 17932288->17936383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.028245:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17932288->17936383] (req 17932288->17936383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.028253:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760696.028259:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17932288->17936383] (req 17932288->17936383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.028269:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17932288->17936383] (req 17932288->17936383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.028293:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17932288->17936383] (req 17932288->17936383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.028302:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207173 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17932288->17936383] (req 17932288->17936383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760696.028526:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484119424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41620us (41867us total) trans 4294971696 rc 0/0 00000100:00100000:0.0:1789760696.028542:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4407 00000100:00100000:0.0:1789760696.028554:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685283442806 time 685317817304 state 0x2 ref 0 00000001:00080000:0.0:1789760696.028606:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971696 is committed 00000100:00100000:2.0:1789760696.034451:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484119552 00000100:00100000:2.0:1789760696.034464:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484119552 00000100:00080000:0.0:1789760696.034532:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760696.034556:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685283442806 time 685317817304 state 0x0 ref 1 00000100:00100000:2.0:1789760696.034576:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685318077720 time 685317817304 state 0x1 ref 1 00000100:00100000:2.0:1789760696.034614:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685318077720 time 685365544148 state 0x0 ref 1 00000100:00100000:2.0:1789760696.034631:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484119552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760696.034713:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17936384, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:2.0:1789760696.034725:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.034757:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:2.0:1789760696.034768:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760696.034776:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17936384->17940479] (req 17936384->17940479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.034793:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17936384->17940479] (req 17936384->17940479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760696.087907:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971697 is committed 00002000:00010000:1.0:1789760696.088019:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17936384 -> 17940480 00010000:00010000:1.0:1789760696.088037:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17936384->17940479] (req 17936384->17940479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.088060:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17936384->17940479] (req 17936384->17940479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.088100:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17936384->17940479] (req 17936384->17940479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.088108:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760696.088114:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17936384->17940479] (req 17936384->17940479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.088124:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17936384->17940479] (req 17936384->17940479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.088150:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17936384->17940479] (req 17936384->17940479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.088158:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620717a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17936384->17940479] (req 17936384->17940479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760696.088326:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484119552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53693us (54033us total) trans 4294971697 rc 0/0 00000100:00100000:1.0:1789760696.088338:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4408 00000100:00100000:1.0:1789760696.088351:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685318077720 time 685365544148 state 0x2 ref 0 00000100:00100000:1.0:1789760696.098429:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:2.0:1789760696.099852:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17940480, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:2.0:1789760696.099866:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.099886:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:2.0:1789760696.099891:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760696.099898:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17940480->17944575] (req 17940480->17944575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.099915:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17940480->17944575] (req 17940480->17944575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760696.136924:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971698 is committed 00002000:00010000:1.0:1789760696.137043:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17940480 -> 17944576 00010000:00010000:1.0:1789760696.137093:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17940480->17944575] (req 17940480->17944575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.137110:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17940480->17944575] (req 17940480->17944575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.137120:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17940480->17944575] (req 17940480->17944575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.137129:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760696.137135:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17940480->17944575] (req 17940480->17944575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.137146:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17940480->17944575] (req 17940480->17944575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.137278:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17940480->17944575] (req 17940480->17944575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.137289:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207181 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17940480->17944575] (req 17940480->17944575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760696.137465:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484119680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38032us (39043us total) trans 4294971698 rc 0/0 00000100:00100000:0.0:1789760696.137478:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4409 00000100:00100000:0.0:1789760696.137490:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685365804564 time 685430336884 state 0x2 ref 0 00000100:00100000:0.0:1789760696.140339:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760696.140534:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484119808 00000100:00100000:1.0:1789760696.140603:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484119808 00000100:00100000:0.0:1789760696.140805:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685365804564 time 685430336884 state 0x0 ref 1 00000100:00100000:0.0:1789760696.140825:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685430597300 time 685430336884 state 0x1 ref 1 00000100:00100000:0.0:1789760696.140881:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685430597300 time 685471811099 state 0x0 ref 1 00000100:00100000:0.0:1789760696.140896:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484119808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760696.140957:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17944576, mtime: 0x6aad94b7, atime: 0x0, ctime: 0x6aad94b7, blocks: 0x9007 00010000:00010000:0.0:1789760696.140967:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.140986:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:0.0:1789760696.140990:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760696.140998:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17944576->17948671] (req 17944576->17948671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.141015:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17944576->17948671] (req 17944576->17948671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760696.188551:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17944576 -> 17948672 00002000:00010000:0.0:1789760696.188564:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760695 -> 1789760696 00002000:00010000:0.0:1789760696.188567:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760695 -> 1789760696 00010000:00010000:0.0:1789760696.188584:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17944576->17948671] (req 17944576->17948671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.188600:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17944576->17948671] (req 17944576->17948671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.188611:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17944576->17948671] (req 17944576->17948671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.188620:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760696.188627:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17944576->17948671] (req 17944576->17948671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760696.188628:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971699 is committed 00010000:00010000:0.0:1789760696.188641:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17944576->17948671] (req 17944576->17948671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.188686:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17944576->17948671] (req 17944576->17948671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.188852:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207188 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17944576->17948671] (req 17944576->17948671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760696.188973:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484119808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48074us (48640us total) trans 4294971699 rc 0/0 00000100:00100000:0.0:1789760696.188986:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4410 00000100:00100000:0.0:1789760696.188998:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685430597300 time 685471811099 state 0x2 ref 0 00000100:00100000:0.0:1789760696.191289:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484119936 00000100:00100000:0.0:1789760696.191302:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484119936 00000100:00100000:0.0:1789760696.191375:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685430597300 time 685471811099 state 0x0 ref 1 00000100:00100000:0.0:1789760696.191394:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685472071515 time 685471811099 state 0x1 ref 1 00000100:00100000:0.0:1789760696.191428:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685472071515 time 685522359184 state 0x0 ref 1 00000100:00100000:0.0:1789760696.191444:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484119936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760696.191507:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17948672, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:0.0:1789760696.191517:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.191536:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:0.0:1789760696.191541:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760696.191548:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17948672->17952767] (req 17948672->17952767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.191570:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17948672->17952767] (req 17948672->17952767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.204282:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760696.249492:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971700 is committed 00002000:00010000:1.0:1789760696.249734:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17948672 -> 17952768 00010000:00010000:1.0:1789760696.249750:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17948672->17952767] (req 17948672->17952767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.249763:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17948672->17952767] (req 17948672->17952767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.249771:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17948672->17952767] (req 17948672->17952767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.249778:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760696.249783:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17948672->17952767] (req 17948672->17952767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.249791:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17948672->17952767] (req 17948672->17952767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.249809:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17948672->17952767] (req 17948672->17952767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.249816:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620718f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17948672->17952767] (req 17948672->17952767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760696.249909:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484119936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58465us (58717us total) trans 4294971700 rc 0/0 00000100:00100000:1.0:1789760696.249920:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4411 00000100:00100000:1.0:1789760696.249929:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685472071515 time 685522359184 state 0x2 ref 0 00000100:00100000:1.0:1789760696.258100:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760696.260415:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484120064 00000100:00100000:2.0:1789760696.260429:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484120064 00000100:00100000:2.0:1789760696.260497:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685472071515 time 685522359184 state 0x0 ref 1 00000100:00100000:2.0:1789760696.260513:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685522619600 time 685522359184 state 0x1 ref 1 00000100:00100000:2.0:1789760696.260550:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685522619600 time 685591481220 state 0x0 ref 1 00000100:00100000:2.0:1789760696.260564:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484120064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760696.260611:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17952768, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:2.0:1789760696.260619:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.260634:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:2.0:1789760696.260638:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760696.260644:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17952768->17956863] (req 17952768->17956863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.260669:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17952768->17956863] (req 17952768->17956863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760696.360042:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17952768 -> 17956864 00010000:00010000:1.0:1789760696.360089:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17952768->17956863] (req 17952768->17956863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.360102:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17952768->17956863] (req 17952768->17956863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.360111:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17952768->17956863] (req 17952768->17956863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.360118:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760696.360123:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17952768->17956863] (req 17952768->17956863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.360133:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17952768->17956863] (req 17952768->17956863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.360153:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17952768->17956863] (req 17952768->17956863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.360161:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207196 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17952768->17956863] (req 17952768->17956863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760696.360186:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971701 is committed 00000100:00100000:1.0:1789760696.360348:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484120064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 99781us (102288us total) trans 4294971701 rc 0/0 00000100:00100000:1.0:1789760696.360358:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4412 00000100:00100000:1.0:1789760696.360368:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685522619600 time 685591481220 state 0x2 ref 0 00000100:00100000:0.0:1789760696.365683:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760696.367282:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484120192 00000100:00100000:2.0:1789760696.367297:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484120192 00000100:00100000:2.0:1789760696.367359:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685522619600 time 685591481220 state 0x0 ref 1 00000100:00100000:2.0:1789760696.367376:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685591741636 time 685591481220 state 0x1 ref 1 00000100:00100000:1.0:1789760696.367488:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685591741636 time 685698416020 state 0x0 ref 1 00000100:00100000:1.0:1789760696.367503:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484120192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760696.367551:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17956864, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:1.0:1789760696.367559:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620719d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.367574:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:1.0:1789760696.367578:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760696.367584:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620719d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17956864->17960959] (req 17956864->17960959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.367597:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620719d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17956864->17960959] (req 17956864->17960959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760696.439584:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971702 is committed 00000100:00100000:2.0:1789760696.444683:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484120192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77175us (79001us total) trans 4294971702 rc 0/0 00000100:00100000:2.0:1789760696.444696:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4413 00000100:00100000:2.0:1789760696.444706:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685591741636 time 685698416020 state 0x2 ref 0 00000100:00100000:0.0:1789760696.450955:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484120320 00000100:00100000:0.0:1789760696.450965:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484120320 00000100:00100000:0.0:1789760696.451028:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685591741636 time 685698416020 state 0x0 ref 1 00000100:00100000:0.0:1789760696.451044:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685698676436 time 685698416020 state 0x1 ref 1 00000100:00100000:0.0:1789760696.451105:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685698676436 time 685782036122 state 0x0 ref 1 00000100:00100000:0.0:1789760696.451120:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7073c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484120320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760696.451169:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17960960, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:0.0:1789760696.451178:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962071a4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.451193:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:0.0:1789760696.451197:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760696.451204:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962071a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17960960->17965055] (req 17960960->17965055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.451217:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962071a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17960960->17965055] (req 17960960->17965055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760696.537911:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971703 is committed 00000100:00100000:2.0:1789760696.552607:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760696.552647:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760696.552684:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760696.552872:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760696.553010:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484120448 00000100:00100000:2.0:1789760696.553022:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484120448 00000100:00100000:2.0:1789760696.553108:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 461 00000100:00100000:2.0:1789760696.553121:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4d2912c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2425:x1876699484120448:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760696.553203:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4d2912c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2425:x1876699484120448:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 85us (600us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760696.553984:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 461 00000100:00100000:2.0:1789760696.554053:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484120576 00000100:00100000:2.0:1789760696.554062:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484120576 00000100:00100000:2.0:1789760696.554138:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484120832 00000100:00100000:2.0:1789760696.554142:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484120832 00000100:00100000:2.0:1789760696.554167:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484120704 00000100:00100000:2.0:1789760696.554169:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484120704 00000100:00100000:2.0:1789760696.554181:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484120960 00000100:00100000:2.0:1789760696.554183:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484120960 00000100:00100000:2.0:1789760696.554198:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 831 00000100:00100000:2.0:1789760696.554205:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484120576:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760696.554957:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484120576:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 749us (3373us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760696.554966:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 831 00000100:00100000:2.0:1789760696.554991:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 832 00000100:00100000:2.0:1789760696.554998:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484120832:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760696.555025:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484120832:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 28us (2377us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760696.555032:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 832 00000100:00100000:2.0:1789760696.555045:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 833 00000100:00100000:2.0:1789760696.555051:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5bb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484120704:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760696.555102:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5bb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484120704:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 50us (2418us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760696.555108:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 833 00000100:00100000:2.0:1789760696.555121:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 834 00000100:00100000:2.0:1789760696.555126:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5ad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484120960:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760696.555151:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5ad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484120960:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 25us (2279us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760696.555157:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 834 00000100:00080000:2.0:1789760696.555193:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760696.555198:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:2.0:1789760696.557554:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484121088 00000100:00100000:2.0:1789760696.557563:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484121088 00000100:00100000:2.0:1789760696.557615:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685698676436 time 685782036122 state 0x0 ref 1 00000100:00100000:2.0:1789760696.557630:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685782296538 time 685782036122 state 0x1 ref 1 00000100:00100000:2.0:1789760696.557844:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685782296538 time 685888773062 state 0x0 ref 1 00000100:00100000:2.0:1789760696.557859:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484121088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760696.557909:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17965056, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:2.0:1789760696.557917:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.557933:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:2.0:1789760696.557938:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760696.557943:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17965056->17969151] (req 17965056->17969151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.557956:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17965056->17969151] (req 17965056->17969151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760696.626631:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17965056 -> 17969152 00010000:00010000:0.0:1789760696.626659:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17965056->17969151] (req 17965056->17969151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.626675:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17965056->17969151] (req 17965056->17969151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.626686:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17965056->17969151] (req 17965056->17969151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.626693:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760696.626699:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17965056->17969151] (req 17965056->17969151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.626709:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17965056->17969151] (req 17965056->17969151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.626733:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17965056->17969151] (req 17965056->17969151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.626800:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962071ab lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17965056->17969151] (req 17965056->17969151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760696.626948:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484121088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69086us (70031us total) trans 4294971704 rc 0/0 00000100:00100000:0.0:1789760696.626963:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4415 00000100:00100000:0.0:1789760696.626976:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685782296538 time 685888773062 state 0x2 ref 0 00000100:00100000:1.0:1789760696.633646:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760696.633780:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484121216 00000100:00100000:2.0:1789760696.633792:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484121216 00000100:00100000:2.0:1789760696.633857:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685782296538 time 685888773062 state 0x0 ref 1 00000100:00100000:2.0:1789760696.633876:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685889033478 time 685888773062 state 0x1 ref 1 00000100:00100000:2.0:1789760696.633952:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685889033478 time 685964873989 state 0x0 ref 1 00000100:00100000:2.0:1789760696.633965:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484121216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760696.634026:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17969152, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:2.0:1789760696.634036:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962071b2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.634055:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=685s v=(0 0) 00010000:00010000:2.0:1789760696.634060:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760696.634103:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962071b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17969152->17973247] (req 17969152->17973247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.634120:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962071b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17969152->17973247] (req 17969152->17973247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760696.672553:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760696.672889:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484121344 00000100:00100000:1.0:1789760696.672910:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484121344 00000100:00100000:1.0:1789760696.672979:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685889033478 time 685964873989 state 0x0 ref 1 00000100:00100000:1.0:1789760696.672998:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685965134405 time 685964873989 state 0x1 ref 1 00000100:00100000:1.0:1789760696.673118:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 685965134405 time 686004048032 state 0x0 ref 1 00000100:00100000:1.0:1789760696.673133:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484121344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760696.673220:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17973248, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:1.0:1789760696.673231:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962071b9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.673250:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:1.0:1789760696.673256:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760696.673263:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962071b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17973248->17977343] (req 17973248->17977343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.673290:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962071b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17973248->17977343] (req 17973248->17977343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760696.717398:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760696.760814:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760696.764045:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484121600 00000100:00100000:2.0:1789760696.764117:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484121600 00000100:00100000:2.0:1789760696.764203:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686004308448 time 686048607750 state 0x0 ref 1 00000100:00100000:2.0:1789760696.764222:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686048868166 time 686048607750 state 0x1 ref 1 00000100:00100000:2.0:1789760696.764261:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686048868166 time 686095191962 state 0x0 ref 1 00000100:00100000:2.0:1789760696.764276:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484121600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760696.764347:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17981440, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:2.0:1789760696.764357:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.764377:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:2.0:1789760696.764382:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760696.764389:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17981440->17985535] (req 17981440->17985535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.764405:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17981440->17985535] (req 17981440->17985535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760696.840413:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17981440 -> 17985536 00010000:00010000:2.0:1789760696.840441:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17981440->17985535] (req 17981440->17985535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.840458:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17981440->17985535] (req 17981440->17985535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.840468:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17981440->17985535] (req 17981440->17985535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.840477:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760696.840483:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17981440->17985535] (req 17981440->17985535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.840493:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17981440->17985535] (req 17981440->17985535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.840517:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17981440->17985535] (req 17981440->17985535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.840525:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962071c7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17981440->17985535] (req 17981440->17985535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760696.840640:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484121600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 76359us (79833us total) trans 4294971708 rc 0/0 00000100:00100000:2.0:1789760696.840661:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4419 00000100:00100000:2.0:1789760696.840673:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686048868166 time 686095191962 state 0x2 ref 0 00000100:00100000:2.0:1789760696.843539:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484121728 00000100:00100000:2.0:1789760696.843552:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484121728 00000100:00100000:2.0:1789760696.843620:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686048868166 time 686095191962 state 0x0 ref 1 00000100:00100000:2.0:1789760696.843638:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686095452378 time 686095191962 state 0x1 ref 1 00000100:00100000:2.0:1789760696.843676:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686095452378 time 686174606178 state 0x0 ref 1 00000100:00100000:2.0:1789760696.843690:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484121728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760696.843760:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 17985536, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:2.0:1789760696.843771:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962071ce lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.843791:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:2.0:1789760696.843796:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760696.843804:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962071ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17985536->17989631] (req 17985536->17989631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760696.843820:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962071ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17985536->17989631] (req 17985536->17989631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760696.874920:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971709 is committed 00000100:00100000:2.0:1789760696.878883:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760696.916836:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971710 is committed 00000100:00100000:1.0:1789760696.920339:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760696.952872:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971711 is committed 00002000:00010000:0.0:1789760696.953053:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17993728 -> 17997824 00010000:00010000:0.0:1789760696.953099:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962071dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17993728->17997823] (req 17993728->17997823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.953116:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962071dc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17993728->17997823] (req 17993728->17997823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.953126:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962071dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17993728->17997823] (req 17993728->17997823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.953134:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760696.953140:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962071dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17993728->17997823] (req 17993728->17997823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.953150:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962071dc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17993728->17997823] (req 17993728->17997823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.953175:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962071dc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17993728->17997823] (req 17993728->17997823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760696.953189:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962071dc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17993728->17997823] (req 17993728->17997823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760696.953314:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484121984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32724us (33035us total) trans 4294971711 rc 0/0 00000100:00100000:0.0:1789760696.953327:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4422 00000100:00100000:0.0:1789760696.953339:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686210400094 time 686251496437 state 0x2 ref 0 00000100:00100000:2.0:1789760696.956862:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760696.956930:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484122112 00000100:00100000:0.0:1789760696.956943:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484122112 00000100:00100000:0.0:1789760696.957023:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686210400094 time 686251496437 state 0x0 ref 1 00000100:00100000:0.0:1789760696.957041:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686251756853 time 686251496437 state 0x1 ref 1 00002000:00010000:1.0:1789760696.992234:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 17997824 -> 18001920 00010000:00010000:1.0:1789760696.992259:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962071e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17997824->18001919] (req 17997824->18001919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.992275:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962071e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17997824->18001919] (req 17997824->18001919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.992296:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962071e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17997824->18001919] (req 17997824->18001919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.992304:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760696.992310:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962071e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17997824->18001919] (req 17997824->18001919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.992327:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962071e3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17997824->18001919] (req 17997824->18001919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.992353:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962071e3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17997824->18001919] (req 17997824->18001919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760696.992361:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962071e3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [17997824->18001919] (req 17997824->18001919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760696.992484:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484122112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35316us (35627us total) trans 4294971712 rc 0/0 00000100:00100000:1.0:1789760696.992497:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4423 00000100:00100000:1.0:1789760696.992508:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686251756853 time 686288030955 state 0x2 ref 0 00000001:00080000:2.0:1789760696.992960:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971712 is committed 00000100:00100000:2.0:1789760696.995270:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760697.028984:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760697.029106:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971713 is committed 00000001:00080000:0.0:1789760697.058267:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971714 is committed 00000100:00100000:0.0:1789760697.061490:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760697.083884:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971715 is committed 00000100:00100000:2.0:1789760697.086876:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760697.086980:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484122624 00000100:00100000:1.0:1789760697.086992:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484122624 00000100:00100000:1.0:1789760697.087061:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686360511555 time 686392694697 state 0x0 ref 1 00000100:00100000:1.0:1789760697.087122:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686392955113 time 686392694697 state 0x1 ref 1 00000100:00100000:1.0:1789760697.087148:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686392955113 time 686418079289 state 0x0 ref 1 00000100:00100000:1.0:1789760697.087162:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484122624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.087230:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18014208, mtime: 0x6aad94b8, atime: 0x0, ctime: 0x6aad94b8, blocks: 0x9007 00010000:00010000:1.0:1789760697.087241:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962071ff lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.087261:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:1.0:1789760697.087266:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.087273:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962071ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18014208->18018303] (req 18014208->18018303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.087299:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962071ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18014208->18018303] (req 18014208->18018303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760697.123145:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971716 is committed 00000100:00100000:2.0:1789760697.126632:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760697.156718:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971717 is committed 00000100:00100000:2.0:1789760697.161937:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760697.186677:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971718 is committed 00002000:00010000:0.0:1789760697.186865:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18022400 -> 18026496 00010000:00010000:0.0:1789760697.186885:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620720d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18022400->18026495] (req 18022400->18026495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.186902:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620720d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18022400->18026495] (req 18022400->18026495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.186913:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620720d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18022400->18026495] (req 18022400->18026495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.186928:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760697.186934:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620720d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18022400->18026495] (req 18022400->18026495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.186944:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620720d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18022400->18026495] (req 18022400->18026495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.186970:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620720d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18022400->18026495] (req 18022400->18026495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.186978:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620720d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18022400->18026495] (req 18022400->18026495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760697.187164:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484122880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24895us (25233us total) trans 4294971718 rc 0/0 00000100:00100000:0.0:1789760697.187176:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4429 00000100:00100000:0.0:1789760697.187189:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686458091157 time 686493182166 state 0x2 ref 0 00000100:00100000:2.0:1789760697.190016:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760697.190253:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484123008 00000100:00100000:0.0:1789760697.190265:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484123008 00000100:00100000:0.0:1789760697.190343:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686458091157 time 686493182166 state 0x0 ref 1 00000100:00100000:0.0:1789760697.190362:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686493442582 time 686493182166 state 0x1 ref 1 00000100:00100000:0.0:1789760697.190437:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686493442582 time 686521359150 state 0x0 ref 1 00000100:00100000:0.0:1789760697.190456:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484123008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760697.190518:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18026496, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:0.0:1789760697.190528:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207214 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.190546:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:0.0:1789760697.190551:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760697.190559:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207214 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18026496->18030591] (req 18026496->18030591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.190575:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207214 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18026496->18030591] (req 18026496->18030591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760697.240221:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:2.0:1789760697.240248:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8f58700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508901376:0@lo:13:osp-pre-0-1.0 00000100:00100000:2.0:1789760697.240332:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760697.240398:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508901376 00000100:00100000:2.0:1789760697.240406:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508901376 00000100:00100000:2.0:1789760697.240465:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 347 00000100:00100000:2.0:1789760697.240473:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508901376:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:2.0:1789760697.240499:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528416768 pending 1703936 free 3839885312 avail 3809476608 00000100:00100000:2.0:1789760697.240543:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508901376:12345-0@lo:13:osp-pre-0-1.0 Request processed in 70us (210us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760697.240550:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 347 00000100:00100000:2.0:1789760697.240598:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8f58700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508901376:0@lo:13:osp-pre-0-1.0 00000100:00080000:2.0:1789760697.240620:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760697.268613:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971719 is committed 00000100:00100000:2.0:1789760697.272256:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760697.272410:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484123136 00000100:00100000:1.0:1789760697.272424:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484123136 00000100:00100000:1.0:1789760697.272503:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686493442582 time 686521359150 state 0x0 ref 1 00000100:00100000:1.0:1789760697.272522:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686521619566 time 686521359150 state 0x1 ref 1 00000100:00100000:1.0:1789760697.272555:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686521619566 time 686603485434 state 0x0 ref 1 00000100:00100000:1.0:1789760697.272570:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484123136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.272629:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18030592, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760697.272639:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.272669:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:1.0:1789760697.272674:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.272681:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18030592->18034687] (req 18030592->18034687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.272700:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18030592->18034687] (req 18030592->18034687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760697.345972:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18030592 -> 18034688 00010000:00010000:0.0:1789760697.345998:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18030592->18034687] (req 18030592->18034687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.346014:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18030592->18034687] (req 18030592->18034687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.346025:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18030592->18034687] (req 18030592->18034687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.346034:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760697.346039:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18030592->18034687] (req 18030592->18034687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.346051:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18030592->18034687] (req 18030592->18034687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760697.346211:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971720 is committed 00010000:00010000:0.0:1789760697.346272:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18030592->18034687] (req 18030592->18034687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.346283:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620721b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18030592->18034687] (req 18030592->18034687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760697.346495:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484123136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73922us (74245us total) trans 4294971720 rc 0/0 00000100:00100000:1.0:1789760697.346508:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4431 00000100:00100000:1.0:1789760697.346521:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686521619566 time 686603485434 state 0x2 ref 0 00000100:00100000:2.0:1789760697.349370:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:1.0:1789760697.386705:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18034688 -> 18038784 00010000:00010000:1.0:1789760697.386725:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207222 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18034688->18038783] (req 18034688->18038783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.386740:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207222 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18034688->18038783] (req 18034688->18038783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.386755:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207222 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18034688->18038783] (req 18034688->18038783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.386764:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760697.386770:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207222 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18034688->18038783] (req 18034688->18038783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.386780:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207222 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18034688->18038783] (req 18034688->18038783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.386810:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207222 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18034688->18038783] (req 18034688->18038783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.386818:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207222 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18034688->18038783] (req 18034688->18038783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760697.386934:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484123264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37233us (37573us total) trans 4294971721 rc 0/0 00000100:00100000:1.0:1789760697.386945:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4432 00000100:00100000:1.0:1789760697.386957:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686603745850 time 686680608906 state 0x2 ref 0 00000100:00100000:2.0:1789760697.389619:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760697.389778:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484123392 00000100:00100000:0.0:1789760697.389790:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484123392 00000100:00100000:0.0:1789760697.389866:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686603745850 time 686680608906 state 0x0 ref 1 00000100:00100000:0.0:1789760697.389883:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686680869322 time 686680608906 state 0x1 ref 1 00000100:00100000:0.0:1789760697.389915:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686680869322 time 686720846043 state 0x0 ref 1 00000100:00100000:0.0:1789760697.389930:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484123392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760697.389990:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18038784, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:0.0:1789760697.390001:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.390019:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:0.0:1789760697.390024:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760697.390031:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18038784->18042879] (req 18038784->18042879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.390049:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18038784->18042879] (req 18038784->18042879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760697.423832:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18038784 -> 18042880 00010000:00010000:0.0:1789760697.423852:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18038784->18042879] (req 18038784->18042879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.423868:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18038784->18042879] (req 18038784->18042879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.423879:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18038784->18042879] (req 18038784->18042879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.423888:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760697.423894:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18038784->18042879] (req 18038784->18042879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.423904:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18038784->18042879] (req 18038784->18042879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.423928:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18038784->18042879] (req 18038784->18042879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.423936:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207229 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18038784->18042879] (req 18038784->18042879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760697.424143:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484123392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34209us (34530us total) trans 4294971722 rc 0/0 00000100:00100000:0.0:1789760697.424156:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4433 00000100:00100000:0.0:1789760697.424176:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686680869322 time 686720846043 state 0x2 ref 0 00000100:00100000:2.0:1789760697.428626:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760697.428732:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484123520 00000100:00100000:0.0:1789760697.428744:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484123520 00000100:00100000:0.0:1789760697.428875:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686680869322 time 686720846043 state 0x0 ref 1 00000100:00100000:0.0:1789760697.428895:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686721106459 time 686720846043 state 0x1 ref 1 00000100:00100000:0.0:1789760697.428961:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686721106459 time 686759891697 state 0x0 ref 1 00000100:00100000:0.0:1789760697.428979:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484123520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760697.429040:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18042880, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:0.0:1789760697.429050:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.429109:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:0.0:1789760697.429115:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760697.429122:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18042880->18046975] (req 18042880->18046975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.429140:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18042880->18046975] (req 18042880->18046975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760697.468400:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971723 is committed 00002000:00010000:1.0:1789760697.468574:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18042880 -> 18046976 00010000:00010000:1.0:1789760697.468594:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18042880->18046975] (req 18042880->18046975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.468610:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18042880->18046975] (req 18042880->18046975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.468620:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18042880->18046975] (req 18042880->18046975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.468628:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760697.468634:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18042880->18046975] (req 18042880->18046975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.468663:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18042880->18046975] (req 18042880->18046975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.468687:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18042880->18046975] (req 18042880->18046975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.468695:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207230 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18042880->18046975] (req 18042880->18046975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760697.468813:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484123520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39832us (40193us total) trans 4294971723 rc 0/0 00000100:00100000:1.0:1789760697.468825:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4434 00000100:00100000:1.0:1789760697.468837:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686721106459 time 686759891697 state 0x2 ref 0 00000100:00100000:2.0:1789760697.471492:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760697.471584:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484123648 00000100:00100000:1.0:1789760697.471597:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484123648 00000100:00100000:1.0:1789760697.471682:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686721106459 time 686759891697 state 0x0 ref 1 00000100:00100000:1.0:1789760697.471700:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686760152113 time 686759891697 state 0x1 ref 1 00000100:00100000:1.0:1789760697.471836:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686760152113 time 686802765723 state 0x0 ref 1 00000100:00100000:1.0:1789760697.471851:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484123648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.471917:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18046976, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760697.471927:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.471948:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:1.0:1789760697.471953:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.471960:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18046976->18051071] (req 18046976->18051071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.471976:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18046976->18051071] (req 18046976->18051071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760697.510901:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971724 is committed 00002000:00010000:1.0:1789760697.511026:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18046976 -> 18051072 00010000:00010000:1.0:1789760697.511046:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18046976->18051071] (req 18046976->18051071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.511063:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18046976->18051071] (req 18046976->18051071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.511103:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18046976->18051071] (req 18046976->18051071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.511112:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760697.511117:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18046976->18051071] (req 18046976->18051071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.511128:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18046976->18051071] (req 18046976->18051071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.511153:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18046976->18051071] (req 18046976->18051071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.511172:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207237 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18046976->18051071] (req 18046976->18051071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760697.511280:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484123648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39425us (39845us total) trans 4294971724 rc 0/0 00000100:00100000:1.0:1789760697.511291:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4435 00000100:00100000:1.0:1789760697.511304:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686760152113 time 686802765723 state 0x2 ref 0 00000100:00100000:2.0:1789760697.514804:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760697.514886:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484123776 00000100:00100000:1.0:1789760697.514899:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484123776 00000100:00100000:1.0:1789760697.514968:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686760152113 time 686802765723 state 0x0 ref 1 00000100:00100000:1.0:1789760697.514986:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686803026139 time 686802765723 state 0x1 ref 1 00000100:00100000:1.0:1789760697.515020:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686803026139 time 686845950195 state 0x0 ref 1 00000100:00100000:1.0:1789760697.515033:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484123776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.515118:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18051072, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760697.515129:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.515147:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:1.0:1789760697.515152:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.515223:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18051072->18055167] (req 18051072->18055167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.515247:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18051072->18055167] (req 18051072->18055167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760697.582222:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18051072 -> 18055168 00010000:00010000:2.0:1789760697.582249:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18051072->18055167] (req 18051072->18055167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760697.582277:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18051072->18055167] (req 18051072->18055167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760697.582289:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18051072->18055167] (req 18051072->18055167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760697.582296:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760697.582302:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18051072->18055167] (req 18051072->18055167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760697.582312:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18051072->18055167] (req 18051072->18055167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760697.582335:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18051072->18055167] (req 18051072->18055167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760697.582342:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620723e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18051072->18055167] (req 18051072->18055167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760697.582740:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484123776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67701us (67942us total) trans 4294971725 rc 0/0 00000100:00100000:2.0:1789760697.582754:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4436 00000100:00100000:2.0:1789760697.582767:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686803026139 time 686845950195 state 0x2 ref 0 00000001:00080000:0.0:1789760697.583004:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971725 is committed 00000100:00100000:2.0:1789760697.589995:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760697.590101:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484123904 00000100:00100000:0.0:1789760697.590114:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484123904 00000100:00100000:0.0:1789760697.590189:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686803026139 time 686845950195 state 0x0 ref 1 00000100:00100000:0.0:1789760697.590206:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686846210611 time 686845950195 state 0x1 ref 1 00000100:00100000:0.0:1789760697.590241:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686846210611 time 686921171527 state 0x0 ref 1 00000100:00100000:0.0:1789760697.590256:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484123904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760697.590459:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18055168, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:0.0:1789760697.590470:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207245 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.590490:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:0.0:1789760697.590495:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760697.590502:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207245 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18055168->18059263] (req 18055168->18059263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.590520:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207245 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18055168->18059263] (req 18055168->18059263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760697.635708:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971726 is committed 00000100:00100000:0.0:1789760697.639748:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484124032 00000100:00100000:0.0:1789760697.639761:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484124032 00000100:00100000:0.0:1789760697.639839:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686846210611 time 686921171527 state 0x0 ref 1 00000100:00100000:0.0:1789760697.639857:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686921431943 time 686921171527 state 0x1 ref 1 00000100:00100000:0.0:1789760697.639886:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686921431943 time 686970816760 state 0x0 ref 1 00000100:00100000:0.0:1789760697.639900:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484124032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760697.639961:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18059264, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:0.0:1789760697.639971:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.639990:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=686s v=(0 0) 00010000:00010000:0.0:1789760697.639995:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760697.640001:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18059264->18063359] (req 18059264->18063359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.640019:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18059264->18063359] (req 18059264->18063359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760697.669018:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18059264 -> 18063360 00010000:00010000:0.0:1789760697.669044:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18059264->18063359] (req 18059264->18063359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.669060:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18059264->18063359] (req 18059264->18063359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.669088:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18059264->18063359] (req 18059264->18063359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.669097:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760697.669103:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18059264->18063359] (req 18059264->18063359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.669113:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18059264->18063359] (req 18059264->18063359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.669135:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18059264->18063359] (req 18059264->18063359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.669144:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620724c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18059264->18063359] (req 18059264->18063359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760697.669264:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484124032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29360us (29688us total) trans 4294971727 rc 0/0 00000100:00100000:0.0:1789760697.669276:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4438 00000100:00100000:0.0:1789760697.669289:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686921431943 time 686970816760 state 0x2 ref 0 00000001:00080000:1.0:1789760697.669366:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971727 is committed 00000100:00100000:2.0:1789760697.674008:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760697.674113:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484124160 00000100:00100000:1.0:1789760697.674125:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484124160 00000100:00100000:1.0:1789760697.674207:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686921431943 time 686970816760 state 0x0 ref 1 00000100:00100000:1.0:1789760697.674225:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686971077176 time 686970816760 state 0x1 ref 1 00000100:00100000:1.0:1789760697.674254:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686971077176 time 687005184912 state 0x0 ref 1 00000100:00100000:1.0:1789760697.674268:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484124160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.674340:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18063360, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760697.674350:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.674373:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760697.674378:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.674385:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18063360->18067455] (req 18063360->18067455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.674404:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18063360->18067455] (req 18063360->18067455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760697.715383:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971728 is committed 00002000:00010000:0.0:1789760697.715690:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18063360 -> 18067456 00010000:00010000:0.0:1789760697.715710:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18063360->18067455] (req 18063360->18067455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.715727:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18063360->18067455] (req 18063360->18067455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.715744:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18063360->18067455] (req 18063360->18067455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.715752:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760697.715758:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18063360->18067455] (req 18063360->18067455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.715768:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18063360->18067455] (req 18063360->18067455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.715791:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18063360->18067455] (req 18063360->18067455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.715800:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207253 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18063360->18067455] (req 18063360->18067455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760697.715915:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484124160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41642us (41914us total) trans 4294971728 rc 0/0 00000100:00100000:0.0:1789760697.715927:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4439 00000100:00100000:0.0:1789760697.715938:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686971077176 time 687005184912 state 0x2 ref 0 00000100:00100000:2.0:1789760697.722907:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760697.723023:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484124288 00000100:00100000:1.0:1789760697.723036:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484124288 00000100:00100000:1.0:1789760697.723147:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 686971077176 time 687005184912 state 0x0 ref 1 00000100:00100000:1.0:1789760697.723179:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687005445328 time 687005184912 state 0x1 ref 1 00000100:00100000:1.0:1789760697.723203:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687005445328 time 687054134722 state 0x0 ref 1 00000100:00100000:1.0:1789760697.723216:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484124288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.723284:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18067456, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760697.723295:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.723317:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760697.723322:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.723329:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18067456->18071551] (req 18067456->18071551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.723347:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18067456->18071551] (req 18067456->18071551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760697.740233:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:2.0:1789760697.740260:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508901504:0@lo:41:osp-pre-1-0.0 00000100:00100000:2.0:1789760697.740347:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760697.740439:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508901504 00000100:00100000:2.0:1789760697.740447:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508901504 00000100:00100000:2.0:1789760697.740499:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 406 00000100:00100000:2.0:1789760697.740508:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508901504:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:2.0:1789760697.740560:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:2.0:1789760697.740606:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508901504:12345-0@lo:41:osp-pre-1-0.0 Request processed in 98us (259us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760697.740614:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 406 00000100:00100000:2.0:1789760697.740677:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508901504:0@lo:41:osp-pre-1-0.0 00002000:00010000:1.0:1789760697.752844:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18067456 -> 18071552 00010000:00010000:1.0:1789760697.752872:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18067456->18071551] (req 18067456->18071551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.752894:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18067456->18071551] (req 18067456->18071551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.752904:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18067456->18071551] (req 18067456->18071551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.752913:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760697.752918:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18067456->18071551] (req 18067456->18071551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.752929:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18067456->18071551] (req 18067456->18071551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.752954:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18067456->18071551] (req 18067456->18071551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.752962:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620725a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18067456->18071551] (req 18067456->18071551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760697.753110:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484124288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29889us (30209us total) trans 4294971729 rc 0/0 00000100:00100000:1.0:1789760697.753122:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4440 00000100:00100000:1.0:1789760697.753134:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687005445328 time 687054134722 state 0x2 ref 0 00000001:00080000:2.0:1789760697.753686:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971729 is committed 00000100:00100000:0.0:1789760697.758018:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484124416 00000100:00100000:0.0:1789760697.758031:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484124416 00000100:00100000:0.0:1789760697.758144:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687005445328 time 687054134722 state 0x0 ref 1 00000100:00100000:0.0:1789760697.758162:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687054395138 time 687054134722 state 0x1 ref 1 00000100:00100000:0.0:1789760697.758191:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687054395138 time 687089121927 state 0x0 ref 1 00000100:00100000:0.0:1789760697.758205:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484124416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760697.758267:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18071552, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:0.0:1789760697.758278:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.758297:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:0.0:1789760697.758302:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760697.758310:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18071552->18075647] (req 18071552->18075647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.758327:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18071552->18075647] (req 18071552->18075647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760697.781630:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971730 is committed 00002000:00010000:1.0:1789760697.781815:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18071552 -> 18075648 00010000:00010000:1.0:1789760697.781835:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18071552->18075647] (req 18071552->18075647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.781851:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18071552->18075647] (req 18071552->18075647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.781862:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18071552->18075647] (req 18071552->18075647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.781871:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760697.781877:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18071552->18075647] (req 18071552->18075647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.781888:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18071552->18075647] (req 18071552->18075647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.781911:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18071552->18075647] (req 18071552->18075647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.781920:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207261 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18071552->18075647] (req 18071552->18075647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760697.782035:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484124416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23825us (24335us total) trans 4294971730 rc 0/0 00000100:00100000:1.0:1789760697.782047:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4441 00000100:00100000:1.0:1789760697.782059:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687054395138 time 687089121927 state 0x2 ref 0 00000100:00100000:2.0:1789760697.785408:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760697.785550:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484124544 00000100:00100000:0.0:1789760697.785565:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484124544 00000100:00100000:0.0:1789760697.785650:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687054395138 time 687089121927 state 0x0 ref 1 00000100:00100000:0.0:1789760697.785670:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687089382343 time 687089121927 state 0x1 ref 1 00000100:00100000:0.0:1789760697.785738:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687089382343 time 687116661599 state 0x0 ref 1 00000100:00100000:0.0:1789760697.785754:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484124544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760697.785816:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18075648, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:0.0:1789760697.785827:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.785848:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:0.0:1789760697.785853:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760697.785861:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18075648->18079743] (req 18075648->18079743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.785877:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18075648->18079743] (req 18075648->18079743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760697.813396:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18075648 -> 18079744 00010000:00010000:1.0:1789760697.813417:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18075648->18079743] (req 18075648->18079743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.813490:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18075648->18079743] (req 18075648->18079743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.813501:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18075648->18079743] (req 18075648->18079743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.813509:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760697.813514:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18075648->18079743] (req 18075648->18079743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.813526:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18075648->18079743] (req 18075648->18079743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.813551:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18075648->18079743] (req 18075648->18079743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.813560:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207268 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18075648->18079743] (req 18075648->18079743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760697.813718:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484124544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27960us (28316us total) trans 4294971731 rc 0/0 00000100:00100000:1.0:1789760697.813738:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4442 00000100:00100000:1.0:1789760697.813750:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687089382343 time 687116661599 state 0x2 ref 0 00000100:00100000:1.0:1789760697.816999:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484124672 00000100:00100000:1.0:1789760697.817012:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484124672 00000100:00100000:1.0:1789760697.817098:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687089382343 time 687116661599 state 0x0 ref 1 00000100:00100000:1.0:1789760697.817117:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687116922015 time 687116661599 state 0x1 ref 1 00000100:00100000:1.0:1789760697.817186:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687116922015 time 687148116568 state 0x0 ref 1 00000100:00100000:1.0:1789760697.817202:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484124672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.817259:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18079744, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760697.817269:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.817288:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760697.817292:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.817300:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18079744->18083839] (req 18079744->18083839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.817316:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18079744->18083839] (req 18079744->18083839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760697.847294:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18079744 -> 18083840 00010000:00010000:1.0:1789760697.847323:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18079744->18083839] (req 18079744->18083839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.847347:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18079744->18083839] (req 18079744->18083839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.847358:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18079744->18083839] (req 18079744->18083839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.847367:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760697.847374:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18079744->18083839] (req 18079744->18083839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.847384:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18079744->18083839] (req 18079744->18083839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.847409:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18079744->18083839] (req 18079744->18083839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.847419:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620726f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18079744->18083839] (req 18079744->18083839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760697.847749:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484124672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30543us (30900us total) trans 4294971732 rc 0/0 00000100:00100000:1.0:1789760697.847762:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4443 00000100:00100000:1.0:1789760697.847774:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687116922015 time 687148116568 state 0x2 ref 0 00000100:00100000:2.0:1789760697.850556:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760697.850780:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484124800 00000100:00100000:1.0:1789760697.850793:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484124800 00000100:00100000:1.0:1789760697.850879:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687116922015 time 687148116568 state 0x0 ref 1 00000100:00100000:1.0:1789760697.850898:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687148376984 time 687148116568 state 0x1 ref 1 00000100:00100000:1.0:1789760697.850963:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687148376984 time 687181892783 state 0x0 ref 1 00000100:00100000:1.0:1789760697.850977:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484124800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.851035:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18083840, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760697.851044:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.851062:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760697.851085:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.851093:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18083840->18087935] (req 18083840->18087935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.851110:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18083840->18087935] (req 18083840->18087935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760697.876027:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971733 is committed 00002000:00010000:1.0:1789760697.876271:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18083840 -> 18087936 00010000:00010000:1.0:1789760697.876293:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18083840->18087935] (req 18083840->18087935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.876308:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18083840->18087935] (req 18083840->18087935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.876325:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18083840->18087935] (req 18083840->18087935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.876334:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760697.876340:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18083840->18087935] (req 18083840->18087935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.876351:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18083840->18087935] (req 18083840->18087935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.876376:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18083840->18087935] (req 18083840->18087935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.876383:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207276 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18083840->18087935] (req 18083840->18087935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760697.876562:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484124800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25580us (26010us total) trans 4294971733 rc 0/0 00000100:00100000:1.0:1789760697.876574:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4444 00000100:00100000:1.0:1789760697.876587:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687148376984 time 687181892783 state 0x2 ref 0 00000100:00100000:1.0:1789760697.879421:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484124928 00000100:00100000:1.0:1789760697.879439:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484124928 00000100:00100000:1.0:1789760697.879505:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687148376984 time 687181892783 state 0x0 ref 1 00000100:00100000:1.0:1789760697.879522:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687182153199 time 687181892783 state 0x1 ref 1 00000100:00100000:1.0:1789760697.879549:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687182153199 time 687210479763 state 0x0 ref 1 00000100:00100000:1.0:1789760697.879561:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484124928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.879621:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18087936, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760697.879631:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.879661:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760697.879666:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.879673:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18087936->18092031] (req 18087936->18092031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.879690:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18087936->18092031] (req 18087936->18092031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760697.925758:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18087936 -> 18092032 00010000:00010000:0.0:1789760697.925783:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18087936->18092031] (req 18087936->18092031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.925801:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18087936->18092031] (req 18087936->18092031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.925811:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18087936->18092031] (req 18087936->18092031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.925820:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760697.925825:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18087936->18092031] (req 18087936->18092031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.925836:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18087936->18092031] (req 18087936->18092031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.925866:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18087936->18092031] (req 18087936->18092031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760697.925875:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620727d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18087936->18092031] (req 18087936->18092031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760697.925988:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484124928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46422us (46689us total) trans 4294971734 rc 0/0 00000100:00100000:0.0:1789760697.926000:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4445 00000100:00100000:0.0:1789760697.926011:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687182153199 time 687210479763 state 0x2 ref 0 00000100:00100000:0.0:1789760697.929140:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484125056 00000100:00100000:0.0:1789760697.929156:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484125056 00000100:00100000:1.0:1789760697.929388:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687182153199 time 687210479763 state 0x0 ref 1 00000100:00100000:1.0:1789760697.929406:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687210740179 time 687210479763 state 0x1 ref 1 00000100:00100000:1.0:1789760697.929441:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687210740179 time 687260372488 state 0x0 ref 1 00000100:00100000:1.0:1789760697.929455:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484125056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760697.929516:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18092032, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760697.929527:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.929547:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760697.929551:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760697.929559:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18092032->18096127] (req 18092032->18096127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760697.929576:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18092032->18096127] (req 18092032->18096127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760698.003474:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18092032 -> 18096128 00010000:00010000:1.0:1789760698.003500:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18092032->18096127] (req 18092032->18096127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.003524:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18092032->18096127] (req 18092032->18096127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.003535:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18092032->18096127] (req 18092032->18096127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.003543:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760698.003548:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18092032->18096127] (req 18092032->18096127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.003559:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18092032->18096127] (req 18092032->18096127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.003583:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18092032->18096127] (req 18092032->18096127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.003593:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207284 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18092032->18096127] (req 18092032->18096127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760698.003718:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484125056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 74259us (74816us total) trans 4294971735 rc 0/0 00000100:00100000:1.0:1789760698.003730:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4446 00000100:00100000:1.0:1789760698.003743:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687210740179 time 687260372488 state 0x2 ref 0 00000100:00100000:2.0:1789760698.009202:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00080000:2.0:1789760698.010754:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:0.0:1789760698.033137:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971736 is committed 00002000:00010000:1.0:1789760698.033247:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18096128 -> 18100224 00010000:00010000:1.0:1789760698.033267:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620728b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18096128->18100223] (req 18096128->18100223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.033282:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620728b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18096128->18100223] (req 18096128->18100223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.033292:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620728b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18096128->18100223] (req 18096128->18100223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.033300:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760698.033305:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620728b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18096128->18100223] (req 18096128->18100223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.033315:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620728b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18096128->18100223] (req 18096128->18100223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.033345:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620728b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18096128->18100223] (req 18096128->18100223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.033353:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620728b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18096128->18100223] (req 18096128->18100223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760698.033552:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484125184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24036us (24358us total) trans 4294971736 rc 0/0 00000100:00100000:1.0:1789760698.033565:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4447 00000100:00100000:1.0:1789760698.033577:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687260632904 time 687340429569 state 0x2 ref 0 00000100:00100000:2.0:1789760698.037566:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760698.037795:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484125312 00000100:00100000:1.0:1789760698.037809:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484125312 00000100:00100000:1.0:1789760698.037880:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687260632904 time 687340429569 state 0x0 ref 1 00000100:00100000:1.0:1789760698.037898:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687340689985 time 687340429569 state 0x1 ref 1 00000100:00100000:1.0:1789760698.037967:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687340689985 time 687368897601 state 0x0 ref 1 00000100:00100000:1.0:1789760698.037981:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484125312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760698.038041:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18100224, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:1.0:1789760698.038051:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.038103:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760698.038110:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760698.038118:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18100224->18104319] (req 18100224->18104319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.038135:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18100224->18104319] (req 18100224->18104319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760698.074771:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971737 is committed 00002000:00010000:0.0:1789760698.075017:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18100224 -> 18104320 00010000:00010000:0.0:1789760698.075037:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18100224->18104319] (req 18100224->18104319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.075052:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18100224->18104319] (req 18100224->18104319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.075062:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18100224->18104319] (req 18100224->18104319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.075096:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760698.075102:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18100224->18104319] (req 18100224->18104319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.075114:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18100224->18104319] (req 18100224->18104319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.075137:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18100224->18104319] (req 18100224->18104319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.075146:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207292 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18100224->18104319] (req 18100224->18104319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760698.075248:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484125312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37263us (37688us total) trans 4294971737 rc 0/0 00000100:00100000:0.0:1789760698.075260:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4448 00000100:00100000:0.0:1789760698.075273:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687340689985 time 687368897601 state 0x2 ref 0 00000100:00100000:2.0:1789760698.078348:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760698.078487:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484125440 00000100:00100000:0.0:1789760698.078501:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484125440 00000100:00100000:0.0:1789760698.078580:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687340689985 time 687368897601 state 0x0 ref 1 00000100:00100000:0.0:1789760698.078598:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687369158017 time 687368897601 state 0x1 ref 1 00000100:00100000:0.0:1789760698.078633:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687369158017 time 687409563751 state 0x0 ref 1 00000100:00100000:0.0:1789760698.078656:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484125440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760698.078717:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18104320, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:0.0:1789760698.078880:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.078903:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:0.0:1789760698.078908:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760698.078915:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18104320->18108415] (req 18104320->18108415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.078933:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18104320->18108415] (req 18104320->18108415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760698.129850:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18104320 -> 18108416 00010000:00010000:0.0:1789760698.130219:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18104320->18108415] (req 18104320->18108415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.130235:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18104320->18108415] (req 18104320->18108415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.130246:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18104320->18108415] (req 18104320->18108415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.130254:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760698.130260:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18104320->18108415] (req 18104320->18108415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.130271:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18104320->18108415] (req 18104320->18108415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.130296:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18104320->18108415] (req 18104320->18108415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.130305:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207299 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18104320->18108415] (req 18104320->18108415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760698.130480:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484125440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51764us (52082us total) trans 4294971738 rc 0/0 00000100:00100000:0.0:1789760698.130493:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4449 00000100:00100000:0.0:1789760698.130506:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687369158017 time 687409563751 state 0x2 ref 0 00000100:00100000:0.0:1789760698.133610:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760698.133712:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484125568 00000100:00100000:2.0:1789760698.133726:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484125568 00000100:00100000:2.0:1789760698.133794:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687369158017 time 687409563751 state 0x0 ref 1 00000100:00100000:2.0:1789760698.133811:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687409824167 time 687409563751 state 0x1 ref 1 00000100:00100000:2.0:1789760698.133876:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687409824167 time 687464806894 state 0x0 ref 1 00000100:00100000:2.0:1789760698.133890:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484125568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760698.133961:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18108416, mtime: 0x6aad94b9, atime: 0x0, ctime: 0x6aad94b9, blocks: 0x9007 00010000:00010000:2.0:1789760698.133971:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.133991:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:2.0:1789760698.133995:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760698.134003:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18108416->18112511] (req 18108416->18112511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.134019:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18108416->18112511] (req 18108416->18112511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760698.179208:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18108416 -> 18112512 00002000:00010000:0.0:1789760698.179214:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760697 -> 1789760698 00002000:00010000:0.0:1789760698.179217:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760697 -> 1789760698 00010000:00010000:0.0:1789760698.179233:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18108416->18112511] (req 18108416->18112511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.179248:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18108416->18112511] (req 18108416->18112511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.179258:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18108416->18112511] (req 18108416->18112511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.179266:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760698.179271:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18108416->18112511] (req 18108416->18112511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.179289:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18108416->18112511] (req 18108416->18112511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.179314:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18108416->18112511] (req 18108416->18112511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.179322:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962072a0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18108416->18112511] (req 18108416->18112511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760698.179441:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484125568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45537us (45827us total) trans 4294971739 rc 0/0 00000100:00100000:0.0:1789760698.179453:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4450 00000100:00100000:0.0:1789760698.179465:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687409824167 time 687464806894 state 0x2 ref 0 00000100:00100000:2.0:1789760698.186118:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760698.186279:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484125696 00000100:00100000:1.0:1789760698.186292:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484125696 00000100:00100000:1.0:1789760698.186362:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687409824167 time 687464806894 state 0x0 ref 1 00000100:00100000:1.0:1789760698.186380:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687465067310 time 687464806894 state 0x1 ref 1 00000100:00100000:1.0:1789760698.186403:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687465067310 time 687517334093 state 0x0 ref 1 00000100:00100000:1.0:1789760698.186416:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484125696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760698.186489:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18112512, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:1.0:1789760698.186499:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962072a7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.186519:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760698.186524:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760698.186531:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962072a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18112512->18116607] (req 18112512->18116607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.186551:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962072a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18112512->18116607] (req 18112512->18116607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760698.214871:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971740 is committed 00000100:00100000:0.0:1789760698.217858:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484125824 00000100:00100000:0.0:1789760698.217870:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484125824 00000100:00100000:0.0:1789760698.217941:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687465067310 time 687517334093 state 0x0 ref 1 00000100:00100000:0.0:1789760698.217957:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687517594509 time 687517334093 state 0x1 ref 1 00000100:00100000:0.0:1789760698.217991:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687517594509 time 687548921964 state 0x0 ref 1 00000100:00100000:0.0:1789760698.218004:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484125824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760698.218099:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18116608, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:0.0:1789760698.218109:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.218129:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:0.0:1789760698.218134:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760698.218141:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18116608->18120703] (req 18116608->18120703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.218158:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18116608->18120703] (req 18116608->18120703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760698.241746:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971741 is committed 00002000:00010000:0.0:1789760698.241870:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18116608 -> 18120704 00010000:00010000:0.0:1789760698.241890:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18116608->18120703] (req 18116608->18120703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.241904:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18116608->18120703] (req 18116608->18120703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.241915:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18116608->18120703] (req 18116608->18120703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.241924:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760698.241930:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18116608->18120703] (req 18116608->18120703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.241940:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18116608->18120703] (req 18116608->18120703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.241980:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18116608->18120703] (req 18116608->18120703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.241989:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072ae lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18116608->18120703] (req 18116608->18120703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760698.242138:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484125824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24128us (24464us total) trans 4294971741 rc 0/0 00000100:00100000:0.0:1789760698.242154:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4452 00000100:00100000:0.0:1789760698.242787:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687517594509 time 687548921964 state 0x2 ref 0 00000100:00100000:2.0:1789760698.245257:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760698.245324:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484125952 00000100:00100000:0.0:1789760698.245337:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484125952 00000100:00100000:0.0:1789760698.245416:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687517594509 time 687548921964 state 0x0 ref 1 00000100:00100000:0.0:1789760698.245434:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687549182380 time 687548921964 state 0x1 ref 1 00000100:00100000:0.0:1789760698.245482:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687549182380 time 687576411610 state 0x0 ref 1 00000100:00100000:0.0:1789760698.245497:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484125952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760698.245570:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18120704, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:0.0:1789760698.245580:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.245600:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:0.0:1789760698.245606:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760698.245614:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18120704->18124799] (req 18120704->18124799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.245635:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18120704->18124799] (req 18120704->18124799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.252987:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:1.0:1789760698.279683:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18120704 -> 18124800 00010000:00010000:1.0:1789760698.279702:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18120704->18124799] (req 18120704->18124799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.279719:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18120704->18124799] (req 18120704->18124799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.279729:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18120704->18124799] (req 18120704->18124799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.279737:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760698.279743:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18120704->18124799] (req 18120704->18124799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.279754:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18120704->18124799] (req 18120704->18124799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.279778:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18120704->18124799] (req 18120704->18124799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.279787:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962072b5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18120704->18124799] (req 18120704->18124799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760698.279932:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484125952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34432us (34679us total) trans 4294971742 rc 0/0 00000100:00100000:1.0:1789760698.279944:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4453 00000100:00100000:1.0:1789760698.279955:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687549182380 time 687576411610 state 0x2 ref 0 00000100:00100000:2.0:1789760698.287403:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760698.287531:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484126080 00000100:00100000:0.0:1789760698.287544:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484126080 00000100:00100000:0.0:1789760698.287619:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687549182380 time 687576411610 state 0x0 ref 1 00000100:00100000:0.0:1789760698.287651:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687576672026 time 687576411610 state 0x1 ref 1 00000100:00100000:0.0:1789760698.287715:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687576672026 time 687618645834 state 0x0 ref 1 00000100:00100000:0.0:1789760698.287740:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484126080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760698.287803:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18124800, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:0.0:1789760698.287821:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.287839:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:0.0:1789760698.287843:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760698.287851:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18124800->18128895] (req 18124800->18128895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.287868:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18124800->18128895] (req 18124800->18128895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760698.324723:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971743 is committed 00002000:00010000:1.0:1789760698.325139:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18124800 -> 18128896 00010000:00010000:1.0:1789760698.325159:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18124800->18128895] (req 18124800->18128895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.325175:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18124800->18128895] (req 18124800->18128895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.325185:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18124800->18128895] (req 18124800->18128895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.325195:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760698.325200:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18124800->18128895] (req 18124800->18128895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.325212:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18124800->18128895] (req 18124800->18128895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.325237:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18124800->18128895] (req 18124800->18128895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.325246:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962072bc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18124800->18128895] (req 18124800->18128895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760698.325404:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484126080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37670us (38009us total) trans 4294971743 rc 0/0 00000100:00100000:1.0:1789760698.325416:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4454 00000100:00100000:1.0:1789760698.325480:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687576672026 time 687618645834 state 0x2 ref 0 00000100:00100000:2.0:1789760698.331310:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760698.365353:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971744 is committed 00000100:00100000:2.0:1789760698.368520:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760698.368651:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484126336 00000100:00100000:0.0:1789760698.368662:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484126336 00000100:00100000:0.0:1789760698.368735:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687618906250 time 687662548971 state 0x0 ref 1 00000100:00100000:0.0:1789760698.368752:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687662809387 time 687662548971 state 0x1 ref 1 00000100:00100000:0.0:1789760698.368784:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687662809387 time 687699714466 state 0x0 ref 1 00000100:00100000:0.0:1789760698.368797:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484126336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760698.368865:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18132992, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:0.0:1789760698.368883:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.368901:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:0.0:1789760698.368906:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760698.368913:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18132992->18137087] (req 18132992->18137087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.368932:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18132992->18137087] (req 18132992->18137087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760698.412903:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971745 is committed 00002000:00010000:0.0:1789760698.413126:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18132992 -> 18137088 00010000:00010000:0.0:1789760698.413146:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18132992->18137087] (req 18132992->18137087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.413168:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18132992->18137087] (req 18132992->18137087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.413178:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18132992->18137087] (req 18132992->18137087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.413187:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760698.413201:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18132992->18137087] (req 18132992->18137087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.413213:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18132992->18137087] (req 18132992->18137087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.413237:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18132992->18137087] (req 18132992->18137087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.413245:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962072ca lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18132992->18137087] (req 18132992->18137087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760698.413495:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484126336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44692us (44979us total) trans 4294971745 rc 0/0 00000100:00100000:0.0:1789760698.413507:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4456 00000100:00100000:0.0:1789760698.413520:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687662809387 time 687699714466 state 0x2 ref 0 00000100:00100000:2.0:1789760698.419031:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760698.419179:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484126464 00000100:00100000:1.0:1789760698.419191:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484126464 00000100:00100000:1.0:1789760698.419255:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687662809387 time 687699714466 state 0x0 ref 1 00000100:00100000:1.0:1789760698.419273:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687699974882 time 687699714466 state 0x1 ref 1 00000100:00100000:1.0:1789760698.419303:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687699974882 time 687750234372 state 0x0 ref 1 00000100:00100000:1.0:1789760698.419317:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484126464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760698.419377:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18137088, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:1.0:1789760698.419388:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.419410:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760698.419415:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760698.419432:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18137088->18141183] (req 18137088->18141183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.419450:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18137088->18141183] (req 18137088->18141183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760698.468615:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971746 is committed 00002000:00010000:0.0:1789760698.468819:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18137088 -> 18141184 00010000:00010000:0.0:1789760698.468840:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18137088->18141183] (req 18137088->18141183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.468855:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18137088->18141183] (req 18137088->18141183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.468865:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18137088->18141183] (req 18137088->18141183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.468874:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760698.468880:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18137088->18141183] (req 18137088->18141183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.468890:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18137088->18141183] (req 18137088->18141183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.468912:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18137088->18141183] (req 18137088->18141183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.468921:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962072d1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18137088->18141183] (req 18137088->18141183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760698.469033:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484126464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49711us (50018us total) trans 4294971746 rc 0/0 00000100:00100000:0.0:1789760698.469044:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4457 00000100:00100000:0.0:1789760698.469055:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687699974882 time 687750234372 state 0x2 ref 0 00000100:00100000:2.0:1789760698.479705:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760698.483603:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484126592 00000100:00100000:1.0:1789760698.483626:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484126592 00000100:00100000:1.0:1789760698.483726:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687699974882 time 687750234372 state 0x0 ref 1 00000100:00100000:1.0:1789760698.483869:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687750494788 time 687750234372 state 0x1 ref 1 00000100:00100000:1.0:1789760698.483921:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687750494788 time 687814841837 state 0x0 ref 1 00000100:00100000:1.0:1789760698.483940:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484126592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760698.484011:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18141184, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:1.0:1789760698.484022:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.484043:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:1.0:1789760698.484048:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760698.484057:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18141184->18145279] (req 18141184->18145279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.484094:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18141184->18145279] (req 18141184->18145279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760698.546153:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971747 is committed 00002000:00010000:1.0:1789760698.546299:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18141184 -> 18145280 00010000:00010000:1.0:1789760698.546318:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18141184->18145279] (req 18141184->18145279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.546339:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18141184->18145279] (req 18141184->18145279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.546350:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18141184->18145279] (req 18141184->18145279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.546358:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760698.546363:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18141184->18145279] (req 18141184->18145279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.546373:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18141184->18145279] (req 18141184->18145279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.546397:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18141184->18145279] (req 18141184->18145279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.546405:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962072d8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18141184->18145279] (req 18141184->18145279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760698.546525:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484126592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62585us (66828us total) trans 4294971747 rc 0/0 00000100:00100000:1.0:1789760698.546594:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4458 00000100:00100000:1.0:1789760698.546607:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687750494788 time 687814841837 state 0x2 ref 0 00000100:00100000:1.0:1789760698.556323:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:1.0:1789760698.635448:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18145280 -> 18149376 00010000:00010000:1.0:1789760698.635476:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962072df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18145280->18149375] (req 18145280->18149375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.635492:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962072df lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18145280->18149375] (req 18145280->18149375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.635502:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962072df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18145280->18149375] (req 18145280->18149375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.635511:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760698.635516:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962072df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18145280->18149375] (req 18145280->18149375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.635527:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962072df lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18145280->18149375] (req 18145280->18149375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.635561:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962072df lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18145280->18149375] (req 18145280->18149375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.635571:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962072df lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18145280->18149375] (req 18145280->18149375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760698.635697:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484126720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79130us (79379us total) trans 4294971748 rc 0/0 00000100:00100000:1.0:1789760698.635709:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4459 00000100:00100000:1.0:1789760698.635722:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687815102253 time 687887475110 state 0x2 ref 0 00000001:00080000:0.0:1789760698.636048:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971748 is committed 00000100:00100000:0.0:1789760698.640739:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760698.641031:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484126848 00000100:00100000:2.0:1789760698.641045:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484126848 00000100:00100000:2.0:1789760698.641268:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687815102253 time 687887475110 state 0x0 ref 1 00000100:00100000:2.0:1789760698.641288:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687887735526 time 687887475110 state 0x1 ref 1 00000100:00100000:2.0:1789760698.641352:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687887735526 time 687972281947 state 0x0 ref 1 00000100:00100000:2.0:1789760698.641368:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484126848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760698.641443:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18149376, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:2.0:1789760698.641454:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.641475:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=687s v=(0 0) 00010000:00010000:2.0:1789760698.641479:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760698.641487:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18149376->18153471] (req 18149376->18153471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.641504:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18149376->18153471] (req 18149376->18153471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760698.688161:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971749 is committed 00002000:00010000:1.0:1789760698.688266:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18149376 -> 18153472 00010000:00010000:1.0:1789760698.688285:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18149376->18153471] (req 18149376->18153471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.688301:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18149376->18153471] (req 18149376->18153471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.688312:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18149376->18153471] (req 18149376->18153471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.688320:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760698.688325:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18149376->18153471] (req 18149376->18153471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.688335:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18149376->18153471] (req 18149376->18153471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.688364:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18149376->18153471] (req 18149376->18153471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.688373:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962072e6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18149376->18153471] (req 18149376->18153471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760698.688575:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484126848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47203us (47842us total) trans 4294971749 rc 0/0 00000100:00100000:1.0:1789760698.688587:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4460 00000100:00100000:1.0:1789760698.688600:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687887735526 time 687972281947 state 0x2 ref 0 00000100:00100000:0.0:1789760698.690809:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484126976 00000100:00100000:0.0:1789760698.690822:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484126976 00000100:00100000:0.0:1789760698.690893:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687887735526 time 687972281947 state 0x0 ref 1 00000100:00100000:0.0:1789760698.690911:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687972542363 time 687972281947 state 0x1 ref 1 00000100:00100000:0.0:1789760698.690941:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687972542363 time 688021871682 state 0x0 ref 1 00000100:00100000:0.0:1789760698.690954:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484126976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760698.691012:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18153472, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:0.0:1789760698.691022:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.691039:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:0.0:1789760698.691044:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760698.691052:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18153472->18157567] (req 18153472->18157567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.691092:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18153472->18157567] (req 18153472->18157567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760698.723501:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18153472 -> 18157568 00010000:00010000:2.0:1789760698.723520:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18153472->18157567] (req 18153472->18157567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.723542:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18153472->18157567] (req 18153472->18157567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.723553:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18153472->18157567] (req 18153472->18157567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.723561:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760698.723566:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18153472->18157567] (req 18153472->18157567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.723577:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18153472->18157567] (req 18153472->18157567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.723603:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18153472->18157567] (req 18153472->18157567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.723611:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962072ed lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18153472->18157567] (req 18153472->18157567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760698.726845:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484126976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35878us (36160us total) trans 4294971750 rc 0/0 00000100:00100000:2.0:1789760698.726865:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4461 00000100:00100000:2.0:1789760698.726877:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687972542363 time 688021871682 state 0x2 ref 0 00000100:00100000:2.0:1789760698.734942:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760698.735228:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484127104 00000100:00100000:1.0:1789760698.735241:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484127104 00000100:00100000:1.0:1789760698.735317:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 687972542363 time 688021871682 state 0x0 ref 1 00000100:00100000:1.0:1789760698.735336:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688022132098 time 688021871682 state 0x1 ref 1 00000100:00100000:1.0:1789760698.735375:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688022132098 time 688066305679 state 0x0 ref 1 00000100:00100000:1.0:1789760698.735389:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484127104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760698.735464:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18157568, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:1.0:1789760698.735475:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962072f4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.735497:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:1.0:1789760698.735503:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760698.735509:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962072f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18157568->18161663] (req 18157568->18161663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760698.735526:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962072f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18157568->18161663] (req 18157568->18161663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760698.789907:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971751 is committed 00000100:00100000:1.0:1789760698.796950:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760698.824303:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971752 is committed 00000100:00100000:0.0:1789760698.830238:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760698.830489:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484127360 00000100:00100000:2.0:1789760698.830508:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484127360 00000100:00100000:2.0:1789760698.830584:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688066566095 time 688128210230 state 0x0 ref 1 00000100:00100000:2.0:1789760698.830603:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688128470646 time 688128210230 state 0x1 ref 1 00000001:00080000:2.0:1789760698.874666:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971753 is committed 00002000:00010000:0.0:1789760698.876287:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18165760 -> 18169856 00010000:00010000:0.0:1789760698.876311:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207302 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18165760->18169855] (req 18165760->18169855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.876327:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207302 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18165760->18169855] (req 18165760->18169855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.876337:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207302 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18165760->18169855] (req 18165760->18169855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.876345:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760698.876351:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207302 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18165760->18169855] (req 18165760->18169855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.876362:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207302 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18165760->18169855] (req 18165760->18169855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.876386:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207302 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18165760->18169855] (req 18165760->18169855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.876394:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207302 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18165760->18169855] (req 18165760->18169855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760698.876515:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484127360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45828us (46285us total) trans 4294971753 rc 0/0 00000100:00100000:0.0:1789760698.876528:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4464 00000100:00100000:0.0:1789760698.876540:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688128470646 time 688161595269 state 0x2 ref 0 00000100:00100000:1.0:1789760698.880274:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760698.880513:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484127488 00000100:00100000:2.0:1789760698.880525:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484127488 00000100:00100000:2.0:1789760698.880590:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688128470646 time 688161595269 state 0x0 ref 1 00000100:00100000:2.0:1789760698.880609:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688161855685 time 688161595269 state 0x1 ref 1 00000100:00100000:2.0:1789760698.880687:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688161855685 time 688211616849 state 0x0 ref 1 00000100:00100000:2.0:1789760698.880701:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484127488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760698.880769:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18169856, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:2.0:1789760698.880780:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.880798:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:2.0:1789760698.880803:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760698.880810:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18169856->18173951] (req 18169856->18173951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.880827:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18169856->18173951] (req 18169856->18173951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760698.914198:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18169856 -> 18173952 00010000:00010000:0.0:1789760698.914224:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18169856->18173951] (req 18169856->18173951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.914240:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18169856->18173951] (req 18169856->18173951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.914251:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18169856->18173951] (req 18169856->18173951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.914259:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760698.914265:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18169856->18173951] (req 18169856->18173951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.914275:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18169856->18173951] (req 18169856->18173951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.914300:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18169856->18173951] (req 18169856->18173951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760698.914308:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207309 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18169856->18173951] (req 18169856->18173951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760698.914428:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484127488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33716us (34154us total) trans 4294971754 rc 0/0 00000100:00100000:0.0:1789760698.914441:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4465 00000100:00100000:0.0:1789760698.914454:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688161855685 time 688211616849 state 0x2 ref 0 00000100:00100000:1.0:1789760698.917657:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760698.970759:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971755 is committed 00000100:00100000:1.0:1789760698.973942:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484127744 00000100:00100000:1.0:1789760698.973956:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484127744 00000100:00100000:1.0:1789760698.974035:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688211877265 time 688249160262 state 0x0 ref 1 00000100:00100000:1.0:1789760698.974055:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688249420678 time 688249160262 state 0x1 ref 1 00000100:00100000:1.0:1789760698.974166:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688249420678 time 688305096365 state 0x0 ref 1 00000100:00100000:1.0:1789760698.974182:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484127744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760698.974344:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18178048, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:2.0:1789760698.974357:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207317 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.974376:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:2.0:1789760698.974381:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760698.974389:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207317 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18178048->18182143] (req 18178048->18182143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760698.974409:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207317 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18178048->18182143] (req 18178048->18182143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760698.999603:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971756 is committed 00000100:00100000:2.0:1789760699.010344:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760699.010454:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484127872 00000100:00100000:0.0:1789760699.010470:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484127872 00000100:00100000:0.0:1789760699.010549:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688249420678 time 688305096365 state 0x0 ref 1 00000100:00100000:0.0:1789760699.010566:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688305356781 time 688305096365 state 0x1 ref 1 00000100:00100000:0.0:1789760699.010598:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688305356781 time 688341528773 state 0x0 ref 1 00000100:00100000:0.0:1789760699.010611:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484127872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760699.010684:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18182144, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:0.0:1789760699.010695:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620731e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760699.010714:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:0.0:1789760699.010728:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760699.010737:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620731e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18182144->18186239] (req 18182144->18186239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760699.010754:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620731e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18182144->18186239] (req 18182144->18186239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760699.060194:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971757 is committed 00000100:00100000:1.0:1789760699.069721:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760699.070430:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484128000 00000100:00100000:0.0:1789760699.070444:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484128000 00000100:00100000:1.0:1789760699.070566:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688305356781 time 688341528773 state 0x0 ref 1 00000100:00100000:1.0:1789760699.070585:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688341789189 time 688341528773 state 0x1 ref 1 00000100:00100000:1.0:1789760699.070621:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688341789189 time 688401551660 state 0x0 ref 1 00000100:00100000:1.0:1789760699.070644:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484128000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760699.070707:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18186240, mtime: 0x6aad94ba, atime: 0x0, ctime: 0x6aad94ba, blocks: 0x9007 00010000:00010000:1.0:1789760699.070718:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207325 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.070738:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:1.0:1789760699.070743:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760699.070750:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207325 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18186240->18190335] (req 18186240->18190335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.070767:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207325 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18186240->18190335] (req 18186240->18190335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760699.095597:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971758 is committed 00000100:00100000:1.0:1789760699.099508:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:1.0:1789760699.128154:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18190336 -> 18194432 00010000:00010000:1.0:1789760699.128179:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620732c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18190336->18194431] (req 18190336->18194431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.128196:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620732c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18190336->18194431] (req 18190336->18194431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.128208:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620732c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18190336->18194431] (req 18190336->18194431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.128217:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760699.128222:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620732c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18190336->18194431] (req 18190336->18194431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.128233:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620732c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18190336->18194431] (req 18190336->18194431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.128258:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620732c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18190336->18194431] (req 18190336->18194431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.128266:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620732c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18190336->18194431] (req 18190336->18194431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760699.128374:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484128128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28374us (28874us total) trans 4294971759 rc 0/0 00000100:00100000:1.0:1789760699.128387:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4470 00000100:00100000:1.0:1789760699.128399:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688401812076 time 688430913788 state 0x2 ref 0 00000001:00080000:0.0:1789760699.128893:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971759 is committed 00000100:00100000:2.0:1789760699.132414:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760699.170087:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971760 is committed 00002000:00010000:2.0:1789760699.170207:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18194432 -> 18198528 00002000:00010000:2.0:1789760699.170214:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760698 -> 1789760699 00002000:00010000:2.0:1789760699.170217:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760698 -> 1789760699 00010000:00010000:2.0:1789760699.170233:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207333 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18194432->18198527] (req 18194432->18198527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.170249:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207333 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18194432->18198527] (req 18194432->18198527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.170260:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207333 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18194432->18198527] (req 18194432->18198527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.170279:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760699.170285:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207333 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18194432->18198527] (req 18194432->18198527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.170296:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207333 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18194432->18198527] (req 18194432->18198527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.170320:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207333 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18194432->18198527] (req 18194432->18198527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.170333:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207333 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18194432->18198527] (req 18194432->18198527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760699.170546:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7070c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484128256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37161us (38140us total) trans 4294971760 rc 0/0 00000100:00100000:2.0:1789760699.170557:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4471 00000100:00100000:2.0:1789760699.170569:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688431174204 time 688464297549 state 0x2 ref 0 00000100:00100000:2.0:1789760699.174123:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760699.201604:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971761 is committed 00002000:00010000:1.0:1789760699.201737:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18198528 -> 18202624 00010000:00010000:1.0:1789760699.201756:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620733a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18198528->18202623] (req 18198528->18202623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.201773:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620733a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18198528->18202623] (req 18198528->18202623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.201783:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620733a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18198528->18202623] (req 18198528->18202623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.201792:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760699.201797:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620733a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18198528->18202623] (req 18198528->18202623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.201807:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620733a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18198528->18202623] (req 18198528->18202623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.201832:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620733a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18198528->18202623] (req 18198528->18202623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.201847:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620733a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18198528->18202623] (req 18198528->18202623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760699.201951:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484128384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27569us (27836us total) trans 4294971761 rc 0/0 00000100:00100000:1.0:1789760699.201964:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4472 00000100:00100000:1.0:1789760699.201980:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688464557965 time 688505294407 state 0x2 ref 0 00000100:00100000:1.0:1789760699.205657:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760699.237178:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971762 is committed 00002000:00010000:1.0:1789760699.237326:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18202624 -> 18206720 00010000:00010000:1.0:1789760699.237347:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207341 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18202624->18206719] (req 18202624->18206719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.237363:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207341 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18202624->18206719] (req 18202624->18206719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.237374:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207341 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18202624->18206719] (req 18202624->18206719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.237382:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760699.237387:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207341 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18202624->18206719] (req 18202624->18206719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.237398:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207341 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18202624->18206719] (req 18202624->18206719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.237429:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207341 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18202624->18206719] (req 18202624->18206719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.237439:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207341 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18202624->18206719] (req 18202624->18206719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760699.237557:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484128512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31497us (31918us total) trans 4294971762 rc 0/0 00000100:00100000:1.0:1789760699.237570:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4473 00000100:00100000:1.0:1789760699.237582:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688505554823 time 688536970392 state 0x2 ref 0 00000100:00100000:2.0:1789760699.240718:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760699.240806:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484128640 00000100:00100000:1.0:1789760699.240820:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484128640 00000100:00100000:1.0:1789760699.240916:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688505554823 time 688536970392 state 0x0 ref 1 00000100:00100000:1.0:1789760699.240936:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688537230808 time 688536970392 state 0x1 ref 1 00000100:00100000:1.0:1789760699.240972:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688537230808 time 688571902435 state 0x0 ref 1 00000100:00100000:1.0:1789760699.240987:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484128640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760699.241060:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18206720, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:1.0:1789760699.241102:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.241123:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:1.0:1789760699.241130:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760699.241138:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18206720->18210815] (req 18206720->18210815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.241156:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18206720->18210815] (req 18206720->18210815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760699.274935:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971763 is committed 00002000:00010000:1.0:1789760699.275058:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18206720 -> 18210816 00010000:00010000:1.0:1789760699.275182:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18206720->18210815] (req 18206720->18210815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.275207:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18206720->18210815] (req 18206720->18210815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.275217:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18206720->18210815] (req 18206720->18210815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.275226:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760699.275233:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18206720->18210815] (req 18206720->18210815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.275243:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18206720->18210815] (req 18206720->18210815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.275276:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18206720->18210815] (req 18206720->18210815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.275290:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207348 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18206720->18210815] (req 18206720->18210815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760699.275408:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484128640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34417us (34697us total) trans 4294971763 rc 0/0 00000100:00100000:1.0:1789760699.275419:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4474 00000100:00100000:1.0:1789760699.275432:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688537230808 time 688571902435 state 0x2 ref 0 00010000:00010000:0.0:1789760699.276862:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760699.278099:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760699.322888:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971764 is committed 00000100:00100000:1.0:1789760699.326316:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760699.382225:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971765 is committed 00002000:00010000:1.0:1789760699.382337:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18214912 -> 18219008 00010000:00010000:1.0:1789760699.382354:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207356 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18214912->18219007] (req 18214912->18219007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.382367:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207356 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18214912->18219007] (req 18214912->18219007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.382376:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207356 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18214912->18219007] (req 18214912->18219007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.382383:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760699.382388:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207356 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18214912->18219007] (req 18214912->18219007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.382398:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207356 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18214912->18219007] (req 18214912->18219007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.382588:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207356 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18214912->18219007] (req 18214912->18219007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.382680:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207356 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18214912->18219007] (req 18214912->18219007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760699.382774:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484128896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56115us (56467us total) trans 4294971765 rc 0/0 00000100:00100000:1.0:1789760699.382785:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4476 00000100:00100000:1.0:1789760699.382795:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688609501874 time 688657572815 state 0x2 ref 0 00000100:00100000:2.0:1789760699.392190:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760699.392286:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484129024 00000100:00100000:1.0:1789760699.392297:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484129024 00000100:00100000:1.0:1789760699.392360:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688609501874 time 688657572815 state 0x0 ref 1 00000100:00100000:1.0:1789760699.392375:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688657833231 time 688657572815 state 0x1 ref 1 00000100:00100000:1.0:1789760699.392403:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688657833231 time 688723334042 state 0x0 ref 1 00000100:00100000:1.0:1789760699.392416:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484129024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760699.392462:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18219008, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:1.0:1789760699.392471:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.392486:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:1.0:1789760699.392490:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760699.392497:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18219008->18223103] (req 18219008->18223103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.392510:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18219008->18223103] (req 18219008->18223103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760699.443321:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971766 is committed 00002000:00010000:1.0:1789760699.444550:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18219008 -> 18223104 00010000:00010000:1.0:1789760699.444568:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18219008->18223103] (req 18219008->18223103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.444581:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18219008->18223103] (req 18219008->18223103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.444590:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18219008->18223103] (req 18219008->18223103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.444598:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760699.444602:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18219008->18223103] (req 18219008->18223103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.444611:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18219008->18223103] (req 18219008->18223103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.444640:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18219008->18223103] (req 18219008->18223103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.444649:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620735d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18219008->18223103] (req 18219008->18223103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760699.444801:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484129024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52382us (52614us total) trans 4294971766 rc 0/0 00000100:00100000:1.0:1789760699.444812:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4477 00000100:00100000:1.0:1789760699.444821:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688657833231 time 688723334042 state 0x2 ref 0 00000100:00100000:2.0:1789760699.455240:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760699.455532:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484129152 00000100:00100000:1.0:1789760699.455546:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484129152 00000100:00100000:1.0:1789760699.455610:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688657833231 time 688723334042 state 0x0 ref 1 00000100:00100000:1.0:1789760699.455624:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688723594458 time 688723334042 state 0x1 ref 1 00000100:00100000:1.0:1789760699.455673:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688723594458 time 688786603414 state 0x0 ref 1 00000100:00100000:1.0:1789760699.455686:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484129152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760699.455736:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18223104, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:1.0:1789760699.455744:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.455760:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:1.0:1789760699.455764:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760699.455769:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18223104->18227199] (req 18223104->18227199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.455782:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18223104->18227199] (req 18223104->18227199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760699.496751:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18223104 -> 18227200 00000001:00080000:0.0:1789760699.496758:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971767 is committed 00010000:00010000:1.0:1789760699.496778:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18223104->18227199] (req 18223104->18227199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.496794:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18223104->18227199] (req 18223104->18227199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.496916:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18223104->18227199] (req 18223104->18227199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.496924:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760699.496930:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18223104->18227199] (req 18223104->18227199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.496940:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18223104->18227199] (req 18223104->18227199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.496967:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18223104->18227199] (req 18223104->18227199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.496975:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207364 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18223104->18227199] (req 18223104->18227199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760699.497272:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484129152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41582us (42042us total) trans 4294971767 rc 0/0 00000100:00100000:1.0:1789760699.497289:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4478 00000100:00100000:1.0:1789760699.497301:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688723594458 time 688786603414 state 0x2 ref 0 00000100:00100000:2.0:1789760699.499709:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484129280 00000100:00100000:2.0:1789760699.499730:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484129280 00000100:00100000:2.0:1789760699.499962:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688723594458 time 688786603414 state 0x0 ref 1 00000100:00100000:2.0:1789760699.499983:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688786863830 time 688786603414 state 0x1 ref 1 00000100:00100000:2.0:1789760699.500050:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688786863830 time 688830980349 state 0x0 ref 1 00000100:00100000:2.0:1789760699.500114:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484129280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760699.500177:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18227200, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:2.0:1789760699.500188:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.500208:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:2.0:1789760699.500214:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760699.500222:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18227200->18231295] (req 18227200->18231295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.500239:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18227200->18231295] (req 18227200->18231295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760699.530913:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971768 is committed 00002000:00010000:2.0:1789760699.531044:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18227200 -> 18231296 00010000:00010000:2.0:1789760699.531098:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18227200->18231295] (req 18227200->18231295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.531116:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18227200->18231295] (req 18227200->18231295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.531126:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18227200->18231295] (req 18227200->18231295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.531134:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760699.531140:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18227200->18231295] (req 18227200->18231295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.531152:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18227200->18231295] (req 18227200->18231295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.531178:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18227200->18231295] (req 18227200->18231295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.531187:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620736b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18227200->18231295] (req 18227200->18231295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760699.531309:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484129280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31191us (31740us total) trans 4294971768 rc 0/0 00000100:00100000:2.0:1789760699.531322:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4479 00000100:00100000:2.0:1789760699.531334:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688786863830 time 688830980349 state 0x2 ref 0 00000100:00100000:2.0:1789760699.534592:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760699.566623:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971769 is committed 00002000:00010000:1.0:1789760699.567779:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18231296 -> 18235392 00010000:00010000:1.0:1789760699.567803:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207372 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18231296->18235391] (req 18231296->18235391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.567820:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207372 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18231296->18235391] (req 18231296->18235391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.567831:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207372 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18231296->18235391] (req 18231296->18235391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.567839:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760699.567955:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207372 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18231296->18235391] (req 18231296->18235391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.567969:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207372 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18231296->18235391] (req 18231296->18235391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.567994:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207372 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18231296->18235391] (req 18231296->18235391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.568004:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207372 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18231296->18235391] (req 18231296->18235391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760699.568230:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484129408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33242us (33644us total) trans 4294971769 rc 0/0 00000100:00100000:1.0:1789760699.568243:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4480 00000100:00100000:1.0:1789760699.568256:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688831240765 time 688865898159 state 0x2 ref 0 00000100:00100000:1.0:1789760699.572041:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760699.572764:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484129536 00000100:00100000:2.0:1789760699.572777:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484129536 00000100:00100000:2.0:1789760699.573915:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688831240765 time 688865898159 state 0x0 ref 1 00000100:00100000:2.0:1789760699.573935:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688866158575 time 688865898159 state 0x1 ref 1 00000100:00100000:2.0:1789760699.573978:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688866158575 time 688904908271 state 0x0 ref 1 00000100:00100000:2.0:1789760699.573996:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484129536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760699.574081:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18235392, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:2.0:1789760699.574092:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.574117:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:2.0:1789760699.574123:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760699.574131:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18235392->18239487] (req 18235392->18239487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.574150:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18235392->18239487] (req 18235392->18239487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760699.603355:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971770 is committed 00002000:00010000:1.0:1789760699.603921:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18235392 -> 18239488 00010000:00010000:1.0:1789760699.603942:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18235392->18239487] (req 18235392->18239487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.603957:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18235392->18239487] (req 18235392->18239487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.603968:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18235392->18239487] (req 18235392->18239487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.603977:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760699.603983:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18235392->18239487] (req 18235392->18239487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.603994:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18235392->18239487] (req 18235392->18239487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.604020:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18235392->18239487] (req 18235392->18239487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.604034:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207379 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18235392->18239487] (req 18235392->18239487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760699.604237:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484129536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30240us (32203us total) trans 4294971770 rc 0/0 00000100:00100000:1.0:1789760699.604249:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4481 00000100:00100000:1.0:1789760699.604261:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688866158575 time 688904908271 state 0x2 ref 0 00000100:00100000:2.0:1789760699.607248:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760699.607341:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484129664 00000100:00100000:1.0:1789760699.607355:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484129664 00000100:00100000:1.0:1789760699.607427:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688866158575 time 688904908271 state 0x0 ref 1 00000100:00100000:1.0:1789760699.607444:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688905168687 time 688904908271 state 0x1 ref 1 00000100:00100000:1.0:1789760699.607476:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688905168687 time 688938407052 state 0x0 ref 1 00000100:00100000:1.0:1789760699.607490:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484129664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760699.607550:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18239488, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:1.0:1789760699.607561:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207380 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.607580:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=688s v=(0 0) 00010000:00010000:1.0:1789760699.607585:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760699.607592:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207380 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18239488->18243583] (req 18239488->18243583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.607614:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207380 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18239488->18243583] (req 18239488->18243583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760699.633517:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971771 is committed 00000100:00100000:2.0:1789760699.636131:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760699.660859:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971772 is committed 00000100:00100000:1.0:1789760699.664964:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760699.686253:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971773 is committed 00000100:00100000:2.0:1789760699.689970:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760699.690190:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484130048 00000100:00100000:1.0:1789760699.690202:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484130048 00000100:00100000:1.0:1789760699.690268:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688967533915 time 688996139946 state 0x0 ref 1 00000100:00100000:1.0:1789760699.690285:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688996400362 time 688996139946 state 0x1 ref 1 00000100:00100000:1.0:1789760699.690342:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688996400362 time 689021272408 state 0x0 ref 1 00000100:00100000:1.0:1789760699.690354:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484130048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760699.690414:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18251776, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:1.0:1789760699.690425:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.690446:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:1.0:1789760699.690450:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760699.690458:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18251776->18255871] (req 18251776->18255871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.690481:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18251776->18255871] (req 18251776->18255871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760699.718735:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18251776 -> 18255872 00010000:00010000:2.0:1789760699.718761:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18251776->18255871] (req 18251776->18255871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.718777:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18251776->18255871] (req 18251776->18255871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.718788:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18251776->18255871] (req 18251776->18255871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.718797:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760699.718803:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18251776->18255871] (req 18251776->18255871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.718813:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18251776->18255871] (req 18251776->18255871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.718838:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18251776->18255871] (req 18251776->18255871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.718846:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207395 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18251776->18255871] (req 18251776->18255871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760699.718968:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484130048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28607us (29002us total) trans 4294971774 rc 0/0 00000100:00100000:2.0:1789760699.718980:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4485 00000100:00100000:2.0:1789760699.718993:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688996400362 time 689021272408 state 0x2 ref 0 00000001:00080000:0.0:1789760699.719645:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971774 is committed 00000100:00100000:2.0:1789760699.723163:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760699.723500:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484130176 00000100:00100000:1.0:1789760699.723514:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484130176 00000100:00100000:1.0:1789760699.723583:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 688996400362 time 689021272408 state 0x0 ref 1 00000100:00100000:1.0:1789760699.723603:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689021532824 time 689021272408 state 0x1 ref 1 00000100:00100000:1.0:1789760699.723646:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689021532824 time 689054576082 state 0x0 ref 1 00000100:00100000:1.0:1789760699.723663:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484130176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760699.723727:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18255872, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:1.0:1789760699.723737:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.723755:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:1.0:1789760699.723760:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760699.723768:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18255872->18259967] (req 18255872->18259967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.723786:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18255872->18259967] (req 18255872->18259967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760699.750686:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971775 is committed 00002000:00010000:2.0:1789760699.750815:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18255872 -> 18259968 00010000:00010000:2.0:1789760699.750836:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18255872->18259967] (req 18255872->18259967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.750852:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18255872->18259967] (req 18255872->18259967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.750862:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18255872->18259967] (req 18255872->18259967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.750871:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760699.750877:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18255872->18259967] (req 18255872->18259967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.750886:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18255872->18259967] (req 18255872->18259967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.750911:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18255872->18259967] (req 18255872->18259967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.750921:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620739c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18255872->18259967] (req 18255872->18259967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760699.751037:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484130176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27373us (27881us total) trans 4294971775 rc 0/0 00000100:00100000:2.0:1789760699.751050:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4486 00000100:00100000:2.0:1789760699.751062:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689021532824 time 689054576082 state 0x2 ref 0 00000100:00100000:2.0:1789760699.753920:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484130304 00000100:00100000:2.0:1789760699.753933:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484130304 00000100:00100000:2.0:1789760699.754001:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689021532824 time 689054576082 state 0x0 ref 1 00000100:00100000:2.0:1789760699.754019:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689054836498 time 689054576082 state 0x1 ref 1 00000100:00100000:2.0:1789760699.754250:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689054836498 time 689084997718 state 0x0 ref 1 00000100:00100000:2.0:1789760699.754266:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484130304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760699.754327:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18259968, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:2.0:1789760699.754338:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.754356:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:2.0:1789760699.754361:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760699.754369:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18259968->18264063] (req 18259968->18264063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.754386:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18259968->18264063] (req 18259968->18264063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760699.799577:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18259968 -> 18264064 00010000:00010000:2.0:1789760699.799598:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18259968->18264063] (req 18259968->18264063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.799613:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18259968->18264063] (req 18259968->18264063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.799623:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18259968->18264063] (req 18259968->18264063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.799640:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760699.799646:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18259968->18264063] (req 18259968->18264063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.799656:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18259968->18264063] (req 18259968->18264063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.799681:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18259968->18264063] (req 18259968->18264063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.799689:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073a3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18259968->18264063] (req 18259968->18264063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760699.799824:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484130304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45553us (46014us total) trans 4294971776 rc 0/0 00000100:00100000:2.0:1789760699.799837:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4487 00000100:00100000:2.0:1789760699.799849:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689054836498 time 689084997718 state 0x2 ref 0 00000100:00100000:1.0:1789760699.806574:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484130432 00000100:00100000:1.0:1789760699.806588:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484130432 00000100:00100000:1.0:1789760699.806682:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689054836498 time 689084997718 state 0x0 ref 1 00000100:00100000:1.0:1789760699.806703:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689085258134 time 689084997718 state 0x1 ref 1 00000100:00100000:1.0:1789760699.806744:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689085258134 time 689137674338 state 0x0 ref 1 00000100:00100000:1.0:1789760699.806760:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484130432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760699.806840:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18264064, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:1.0:1789760699.806851:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.806872:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:1.0:1789760699.806878:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760699.806886:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18264064->18268159] (req 18264064->18268159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760699.806903:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18264064->18268159] (req 18264064->18268159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760699.896673:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18264064 -> 18268160 00010000:00010000:2.0:1789760699.896694:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18264064->18268159] (req 18264064->18268159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.896717:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18264064->18268159] (req 18264064->18268159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.896725:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18264064->18268159] (req 18264064->18268159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.896732:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760699.896737:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18264064->18268159] (req 18264064->18268159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.896745:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18264064->18268159] (req 18264064->18268159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.896765:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18264064->18268159] (req 18264064->18268159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.896773:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962073aa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18264064->18268159] (req 18264064->18268159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760699.896880:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484130432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 90118us (90438us total) trans 4294971777 rc 0/0 00000100:00100000:2.0:1789760699.896892:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4488 00000100:00100000:2.0:1789760699.896902:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689085258134 time 689137674338 state 0x2 ref 0 00000100:00100000:1.0:1789760699.901017:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760699.901235:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484130560 00000100:00100000:2.0:1789760699.901250:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484130560 00000100:00100000:2.0:1789760699.901319:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689085258134 time 689137674338 state 0x0 ref 1 00000100:00100000:2.0:1789760699.901334:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689137934754 time 689137674338 state 0x1 ref 1 00000100:00100000:2.0:1789760699.901362:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689137934754 time 689232293415 state 0x0 ref 1 00000100:00100000:2.0:1789760699.901375:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484130560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760699.901424:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18268160, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:2.0:1789760699.901433:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.901449:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:2.0:1789760699.901453:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760699.901459:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18268160->18272255] (req 18268160->18272255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.901472:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18268160->18272255] (req 18268160->18272255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760699.949368:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18268160 -> 18272256 00010000:00010000:2.0:1789760699.949398:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18268160->18272255] (req 18268160->18272255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.949413:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18268160->18272255] (req 18268160->18272255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.949424:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18268160->18272255] (req 18268160->18272255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.949432:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760699.949437:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18268160->18272255] (req 18268160->18272255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.949446:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18268160->18272255] (req 18268160->18272255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.949471:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18268160->18272255] (req 18268160->18272255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760699.949479:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073b1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18268160->18272255] (req 18268160->18272255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760699.949606:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484130560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48226us (48597us total) trans 4294971778 rc 0/0 00000100:00100000:2.0:1789760699.949619:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4489 00000100:00100000:2.0:1789760699.949641:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689137934754 time 689232293415 state 0x2 ref 0 00000001:00080000:0.0:1789760699.950304:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971778 is committed 00000100:00100000:2.0:1789760699.954871:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484130688 00000100:00100000:2.0:1789760699.954884:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484130688 00000100:00100000:2.0:1789760699.954959:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689137934754 time 689232293415 state 0x0 ref 1 00000100:00100000:2.0:1789760699.954977:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689232553831 time 689232293415 state 0x1 ref 1 00000100:00100000:2.0:1789760699.955013:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689232553831 time 689285943856 state 0x0 ref 1 00000100:00100000:2.0:1789760699.955027:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484130688:12345-192.168.203.10@tcp:4:dd.0 00000001:00080000:0.0:1789760700.002107:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971779 is committed 00000100:00100000:2.0:1789760700.006353:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484130816 00000100:00100000:2.0:1789760700.006366:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484130816 00000100:00100000:2.0:1789760700.006537:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689232553831 time 689285943856 state 0x0 ref 1 00000100:00100000:2.0:1789760700.006555:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689286204272 time 689285943856 state 0x1 ref 1 00000100:00100000:2.0:1789760700.006589:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689286204272 time 689337519860 state 0x0 ref 1 00000100:00100000:2.0:1789760700.006603:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484130816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760700.006674:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18276352, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:2.0:1789760700.006685:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073bf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.006705:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:2.0:1789760700.006711:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760700.006774:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18276352->18280447] (req 18276352->18280447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.006792:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962073bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18276352->18280447] (req 18276352->18280447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760700.042703:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971780 is committed 00000100:00100000:1.0:1789760700.046911:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760700.048991:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484130944 00000100:00100000:0.0:1789760700.049011:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484130944 00000100:00100000:0.0:1789760700.049337:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689286204272 time 689337519860 state 0x0 ref 1 00000100:00100000:0.0:1789760700.049359:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689337780276 time 689337519860 state 0x1 ref 1 00000100:00100000:1.0:1789760700.049542:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689337780276 time 689380469921 state 0x0 ref 1 00000100:00100000:1.0:1789760700.049561:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484130944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760700.049650:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18280448, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:1.0:1789760700.049661:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.049687:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:1.0:1789760700.049692:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760700.049699:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18280448->18284543] (req 18280448->18284543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.049725:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18280448->18284543] (req 18280448->18284543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760700.131297:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18280448 -> 18284544 00010000:00010000:1.0:1789760700.131325:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18280448->18284543] (req 18280448->18284543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.131341:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18280448->18284543] (req 18280448->18284543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.131351:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18280448->18284543] (req 18280448->18284543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.131359:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760700.131365:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18280448->18284543] (req 18280448->18284543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.131375:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18280448->18284543] (req 18280448->18284543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.131400:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18280448->18284543] (req 18280448->18284543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.131478:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962073c6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18280448->18284543] (req 18280448->18284543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760700.131604:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484130944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 82043us (84701us total) trans 4294971781 rc 0/0 00000100:00100000:1.0:1789760700.131617:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4492 00000100:00100000:1.0:1789760700.131639:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689337780276 time 689380469921 state 0x2 ref 0 00000100:00100000:0.0:1789760700.136443:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760700.136499:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484131072 00000100:00100000:1.0:1789760700.136512:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484131072 00000100:00100000:1.0:1789760700.136585:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689337780276 time 689380469921 state 0x0 ref 1 00000100:00100000:1.0:1789760700.136604:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689380730337 time 689380469921 state 0x1 ref 1 00000100:00100000:1.0:1789760700.136649:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689380730337 time 689467579602 state 0x0 ref 1 00000100:00100000:1.0:1789760700.136665:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484131072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760700.136738:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18284544, mtime: 0x6aad94bb, atime: 0x0, ctime: 0x6aad94bb, blocks: 0x9007 00010000:00010000:1.0:1789760700.136749:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.136769:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:1.0:1789760700.136774:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760700.136781:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18284544->18288639] (req 18284544->18288639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.136804:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18284544->18288639] (req 18284544->18288639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760700.179683:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18284544 -> 18288640 00002000:00010000:1.0:1789760700.179699:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760699 -> 1789760700 00002000:00010000:1.0:1789760700.179702:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760699 -> 1789760700 00010000:00010000:1.0:1789760700.179726:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18284544->18288639] (req 18284544->18288639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.179742:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18284544->18288639] (req 18284544->18288639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.179753:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18284544->18288639] (req 18284544->18288639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.179761:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760700.179767:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18284544->18288639] (req 18284544->18288639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.179777:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18284544->18288639] (req 18284544->18288639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.179803:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18284544->18288639] (req 18284544->18288639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.179817:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962073cd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18284544->18288639] (req 18284544->18288639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760700.179943:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484131072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43276us (43501us total) trans 4294971782 rc 0/0 00000100:00100000:1.0:1789760700.179955:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4493 00000100:00100000:1.0:1789760700.179967:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689380730337 time 689467579602 state 0x2 ref 0 00000100:00100000:2.0:1789760700.183125:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760700.218679:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971783 is committed 00000100:00100000:2.0:1789760700.221924:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760700.255530:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971784 is committed 00000100:00100000:0.0:1789760700.258851:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760700.258998:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484131456 00000100:00100000:2.0:1789760700.259011:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484131456 00000100:00100000:2.0:1789760700.259106:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689514532395 time 689553086691 state 0x0 ref 1 00000100:00100000:2.0:1789760700.259125:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689553347107 time 689553086691 state 0x1 ref 1 00000100:00100000:2.0:1789760700.259154:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689553347107 time 689590084419 state 0x0 ref 1 00000100:00100000:2.0:1789760700.259168:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484131456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760700.259231:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18296832, mtime: 0x6aad94bc, atime: 0x0, ctime: 0x6aad94bc, blocks: 0x9007 00010000:00010000:2.0:1789760700.259241:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.259262:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:2.0:1789760700.259267:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760700.259275:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18296832->18300927] (req 18296832->18300927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.259291:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18296832->18300927] (req 18296832->18300927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760700.286609:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18296832 -> 18300928 00000001:00080000:1.0:1789760700.286611:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971785 is committed 00010000:00010000:2.0:1789760700.286643:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18296832->18300927] (req 18296832->18300927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.286666:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18296832->18300927] (req 18296832->18300927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.286677:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18296832->18300927] (req 18296832->18300927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.286685:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760700.286690:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18296832->18300927] (req 18296832->18300927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.286700:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18296832->18300927] (req 18296832->18300927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.286733:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18296832->18300927] (req 18296832->18300927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.286742:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962073e2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18296832->18300927] (req 18296832->18300927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760700.286867:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484131456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27695us (28022us total) trans 4294971785 rc 0/0 00000100:00100000:2.0:1789760700.286879:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4496 00000100:00100000:2.0:1789760700.286892:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689553347107 time 689590084419 state 0x2 ref 0 00000100:00100000:2.0:1789760700.289943:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760700.290338:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484131584 00000100:00100000:1.0:1789760700.290352:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484131584 00000100:00100000:1.0:1789760700.290440:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689553347107 time 689590084419 state 0x0 ref 1 00000100:00100000:1.0:1789760700.290459:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689590344835 time 689590084419 state 0x1 ref 1 00000100:00100000:1.0:1789760700.290494:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689590344835 time 689621424518 state 0x0 ref 1 00000100:00100000:1.0:1789760700.290507:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484131584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760700.290581:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18300928, mtime: 0x6aad94bc, atime: 0x0, ctime: 0x6aad94bc, blocks: 0x9007 00010000:00010000:1.0:1789760700.290591:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.290610:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:1.0:1789760700.290616:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760700.290632:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18300928->18305023] (req 18300928->18305023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.290654:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18300928->18305023] (req 18300928->18305023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.302837:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:2.0:1789760700.324200:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18300928 -> 18305024 00010000:00010000:2.0:1789760700.324221:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18300928->18305023] (req 18300928->18305023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.324238:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18300928->18305023] (req 18300928->18305023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.324249:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18300928->18305023] (req 18300928->18305023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.324257:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760700.324262:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18300928->18305023] (req 18300928->18305023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.324274:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18300928->18305023] (req 18300928->18305023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.324299:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18300928->18305023] (req 18300928->18305023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.324307:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962073e9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18300928->18305023] (req 18300928->18305023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760700.324572:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484131584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34059us (34634us total) trans 4294971786 rc 0/0 00000100:00100000:2.0:1789760700.324587:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4497 00000100:00100000:2.0:1789760700.324602:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689590344835 time 689621424518 state 0x2 ref 0 00000100:00100000:1.0:1789760700.328377:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760700.359895:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971787 is committed 00000100:00100000:0.0:1789760700.365018:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760700.412332:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971788 is committed 00000100:00100000:2.0:1789760700.416136:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760700.416242:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484131968 00000100:00100000:1.0:1789760700.416254:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484131968 00000100:00100000:1.0:1789760700.416329:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689659945240 time 689696298590 state 0x0 ref 1 00000100:00100000:1.0:1789760700.416347:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689696559006 time 689696298590 state 0x1 ref 1 00000100:00100000:1.0:1789760700.416417:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689696559006 time 689747346804 state 0x0 ref 1 00000100:00100000:1.0:1789760700.416432:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484131968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760700.416499:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18313216, mtime: 0x6aad94bc, atime: 0x0, ctime: 0x6aad94bc, blocks: 0x9007 00010000:00010000:1.0:1789760700.416509:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.416527:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:1.0:1789760700.416532:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760700.416540:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18313216->18317311] (req 18313216->18317311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.416556:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18313216->18317311] (req 18313216->18317311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760700.461459:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18313216 -> 18317312 00010000:00010000:1.0:1789760700.461480:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18313216->18317311] (req 18313216->18317311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.461496:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18313216->18317311] (req 18313216->18317311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.461507:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18313216->18317311] (req 18313216->18317311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.461514:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760700.461518:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18313216->18317311] (req 18313216->18317311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.461526:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18313216->18317311] (req 18313216->18317311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.461546:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18313216->18317311] (req 18313216->18317311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760700.461553:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962073fe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18313216->18317311] (req 18313216->18317311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760700.461687:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484131968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45249us (45618us total) trans 4294971789 rc 0/0 00000100:00100000:1.0:1789760700.462005:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4500 00000100:00100000:1.0:1789760700.462443:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689696559006 time 689747346804 state 0x2 ref 0 00000100:00100000:2.0:1789760700.464472:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760700.464568:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484132096 00000100:00100000:0.0:1789760700.464581:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484132096 00000100:00100000:0.0:1789760700.464910:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689696559006 time 689747346804 state 0x0 ref 1 00000100:00100000:0.0:1789760700.464932:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689747607220 time 689747346804 state 0x1 ref 1 00000100:00100000:0.0:1789760700.464967:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689747607220 time 689795897619 state 0x0 ref 1 00000100:00100000:0.0:1789760700.464989:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484132096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760700.465996:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18317312, mtime: 0x6aad94bc, atime: 0x0, ctime: 0x6aad94bc, blocks: 0x9007 00010000:00010000:0.0:1789760700.466011:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207405 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.466032:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=689s v=(0 0) 00010000:00010000:0.0:1789760700.466037:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760700.466045:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207405 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18317312->18321407] (req 18317312->18321407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.466061:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207405 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18317312->18321407] (req 18317312->18321407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760700.514173:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971790 is committed 00000100:00100000:0.0:1789760700.518399:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:2.0:1789760700.599939:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18321408 -> 18325504 00010000:00010000:2.0:1789760700.599959:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620740c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18321408->18325503] (req 18321408->18325503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.599974:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620740c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18321408->18325503] (req 18321408->18325503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.599986:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620740c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18321408->18325503] (req 18321408->18325503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.599995:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760700.600001:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620740c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18321408->18325503] (req 18321408->18325503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.600011:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620740c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18321408->18325503] (req 18321408->18325503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.600035:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620740c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18321408->18325503] (req 18321408->18325503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.600044:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620740c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18321408->18325503] (req 18321408->18325503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760700.600187:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484132224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 81472us (81796us total) trans 4294971791 rc 0/0 00000100:00100000:2.0:1789760700.600200:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4502 00000100:00100000:2.0:1789760700.600213:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689796158035 time 689849627148 state 0x2 ref 0 00000100:00100000:2.0:1789760700.605596:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760700.702037:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484132480 00000100:00100000:1.0:1789760700.702054:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484132480 00000100:00100000:2.0:1789760700.704226:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689849887564 time 689936823128 state 0x0 ref 1 00000100:00100000:2.0:1789760700.704252:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689937083544 time 689936823128 state 0x1 ref 1 00000100:00100000:2.0:1789760700.704427:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689937083544 time 690035356456 state 0x0 ref 1 00000100:00100000:2.0:1789760700.704446:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484132480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760700.704517:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18329600, mtime: 0x6aad94bc, atime: 0x0, ctime: 0x6aad94bc, blocks: 0x9007 00010000:00010000:2.0:1789760700.704527:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.704548:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:2.0:1789760700.704554:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760700.704562:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18329600->18333695] (req 18329600->18333695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760700.704578:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18329600->18333695] (req 18329600->18333695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760700.818339:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508901888:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.818454:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760700.818528:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508902400:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.818558:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760700.818585:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8f5a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508902912:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.818606:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760700.818648:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [3->2] 00000100:00100000:2.0:1789760700.818654:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0001_UUID:3480:1876699508902016:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.818677:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760700.818694:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508902528:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.818989:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760700.819010:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8f59c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508903040:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.819033:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760700.819613:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508902528 00000100:00100000:0.0:1789760700.819633:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508902528 00000100:00100000:0.0:1789760700.819732:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508903040 00000100:00100000:0.0:1789760700.819735:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508903040 00000100:00100000:0.0:1789760700.819764:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 295 00000100:00100000:0.0:1789760700.819774:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508902528:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760700.819849:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508902528:12345-0@lo:400:kworker.0 Request processed in 76us (860us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760700.819857:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 295 00000100:00100000:0.0:1789760700.819876:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 296 00000100:00100000:0.0:1789760700.819884:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508903040:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760700.819914:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508903040:12345-0@lo:400:kworker.0 Request processed in 32us (879us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760700.819920:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 296 00000100:00080000:0.0:1789760700.819939:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:0.0:1789760700.819944:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760700.819988:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508902528:0@lo:400:kworker.0 00000100:00100000:0.0:1789760700.820001:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8f59c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508903040:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.820969:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508901632:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.821038:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760700.821121:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508902144:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.821154:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760700.821181:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508902656:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.821221:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760700.821239:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8f5b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508903168:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.821262:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760700.821287:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:1.0:1789760700.821294:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508901760:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.821322:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760700.821342:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508902272:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.821364:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760700.821383:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8f58e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508902784:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.821406:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760700.821423:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c886d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508903296:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760700.821446:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760700.821487:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508902400 00000100:00100000:1.0:1789760700.821495:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508902400 00000100:00100000:1.0:1789760700.821543:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508902912 00000100:00100000:1.0:1789760700.821546:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508902912 00000100:00100000:1.0:1789760700.821560:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508902016 00000100:00100000:1.0:1789760700.821563:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508902016 00000100:00100000:1.0:1789760700.821573:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508901760 00000100:00100000:1.0:1789760700.821575:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508901760 00000100:00100000:1.0:1789760700.821584:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508902272 00000100:00100000:1.0:1789760700.821586:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508902272 00000100:00100000:1.0:1789760700.821599:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508902784 00000100:00100000:1.0:1789760700.821601:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508902784 00000100:00100000:1.0:1789760700.821616:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 835 00000100:00100000:1.0:1789760700.821635:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508902400:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.821692:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508902400:12345-0@lo:400:kworker.0 Request processed in 57us (3133us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760700.821715:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 835 00000100:00100000:1.0:1789760700.821991:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 836 00000100:00100000:1.0:1789760700.821999:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508902912:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.822035:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508902912:12345-0@lo:400:kworker.0 Request processed in 38us (3429us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760700.822041:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 836 00000100:00100000:1.0:1789760700.822053:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 837 00000100:00100000:1.0:1789760700.822059:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3480:x1876699508902016:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.822120:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3480:x1876699508902016:12345-0@lo:400:kworker.0 Request processed in 61us (3442us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760700.822126:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 837 00000100:00100000:1.0:1789760700.822138:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 838 00000100:00100000:1.0:1789760700.822144:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508901760:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.822179:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508901760:12345-0@lo:400:kworker.0 Request processed in 35us (857us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760700.822187:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 838 00000100:00100000:1.0:1789760700.822202:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 839 00000100:00100000:1.0:1789760700.822208:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb726d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508902272:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.822234:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb726d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508902272:12345-0@lo:400:kworker.0 Request processed in 27us (869us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760700.822240:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 839 00000100:00100000:1.0:1789760700.822251:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 840 00000100:00100000:1.0:1789760700.822257:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508902784:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.822282:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508902784:12345-0@lo:400:kworker.0 Request processed in 26us (875us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760700.822288:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 840 00000100:00100000:1.0:1789760700.822343:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508901760:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.822355:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508902272:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.822364:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8f58e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508902784:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.823544:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508902656 00000100:00100000:2.0:1789760700.823558:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508902656 00000100:00100000:2.0:1789760700.823609:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508903168 00000100:00100000:2.0:1789760700.823612:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508903168 00000100:00100000:2.0:1789760700.823659:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 297 00000100:00100000:2.0:1789760700.823667:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508902656:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.824646:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508902656:12345-0@lo:400:kworker.0 Request processed in 973us (3428us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760700.824660:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 297 00000100:00100000:2.0:1789760700.824691:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 298 00000100:00100000:0.0:1789760700.825269:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508901632 00000100:00100000:0.0:1789760700.825282:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508901632 00000100:00100000:0.0:1789760700.825339:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 462 00000100:00100000:0.0:1789760700.825348:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4d2913c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508901632:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760700.825429:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4d2913c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508901632:12345-0@lo:400:kworker.0 Request processed in 83us (4391us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760700.825437:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 462 00000100:00080000:0.0:1789760700.825488:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1789760700.827007:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508903168:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.827152:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508903168:12345-0@lo:400:kworker.0 Request processed in 2453us (5888us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760700.827163:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 298 00000100:00100000:2.0:1789760700.827217:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508901888 00000100:00100000:2.0:1789760700.827227:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508901888 00000100:00100000:2.0:1789760700.827271:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508902144 00000100:00100000:2.0:1789760700.827275:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508902144 00000100:00100000:2.0:1789760700.827291:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 407 00000100:00100000:2.0:1789760700.827298:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58000 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508901888:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.827339:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58000 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508901888:12345-0@lo:400:kworker.0 Request processed in 40us (8886us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760700.827345:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 407 00000100:00100000:2.0:1789760700.827357:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 408 00000100:00100000:2.0:1789760700.827363:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508902144:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.827394:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508902144:12345-0@lo:400:kworker.0 Request processed in 31us (6239us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760700.827406:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 408 00000100:00100000:2.0:1789760700.827669:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508901632:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.827687:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508902144:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.827701:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508902656:0@lo:400:kworker.0 00000100:00100000:2.0:1789760700.827713:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8f5b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508903168:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.828604:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0001_UUID:3480:1876699508902016:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.828634:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508901888:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.828650:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508902400:0@lo:400:kworker.0 00000100:00100000:1.0:1789760700.828664:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8f5a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508902912:0@lo:400:kworker.0 00000100:00080000:0.0:1789760700.838445:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:0.0:1789760700.838459:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:0.0:1789760700.838464:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:2.0:1789760700.840103:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508903424 00000100:00100000:2.0:1789760700.840126:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508903424 00000100:00100000:2.0:1789760700.840249:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 348 00000100:00100000:2.0:1789760700.840263:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508903424:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:2.0:1789760700.840318:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760700.840446:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508903424:12345-0@lo:13:osp-pre-1-1.0 Request processed in 186us (11121us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760700.840455:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 348 00000100:00100000:1.0:1789760700.840557:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f44000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508903424:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760700.844254:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 349 00000100:00100000:2.0:1789760700.844283:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb726680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508903296:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1789760700.844326:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760700.844450:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb726680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508903296:12345-0@lo:13:osp-pre-1-0.0 Request processed in 181us (23001us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760700.844459:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 349 00000100:00100000:2.0:1789760700.846314:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c886d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508903296:0@lo:13:osp-pre-1-0.0 00002000:00010000:0.0:1789760700.857664:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18329600 -> 18333696 00010000:00010000:0.0:1789760700.857692:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18329600->18333695] (req 18329600->18333695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.857994:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18329600->18333695] (req 18329600->18333695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.858005:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18329600->18333695] (req 18329600->18333695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.858023:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760700.858029:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18329600->18333695] (req 18329600->18333695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.858040:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18329600->18333695] (req 18329600->18333695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.858089:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18329600->18333695] (req 18329600->18333695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.858099:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620741a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18329600->18333695] (req 18329600->18333695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760700.858229:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484132480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 153783us (157691us total) trans 4294971793 rc 0/0 00000100:00100000:0.0:1789760700.858242:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4504 00000100:00100000:0.0:1789760700.858255:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 689937083544 time 690035356456 state 0x2 ref 0 00000100:00100000:0.0:1789760700.862843:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760700.907612:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971794 is committed 00002000:00010000:0.0:1789760700.907771:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18333696 -> 18337792 00010000:00010000:0.0:1789760700.907798:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207421 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18333696->18337791] (req 18333696->18337791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.907813:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207421 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18333696->18337791] (req 18333696->18337791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.907823:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207421 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18333696->18337791] (req 18333696->18337791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.907831:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760700.907837:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207421 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18333696->18337791] (req 18333696->18337791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.907854:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207421 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18333696->18337791] (req 18333696->18337791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.907880:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207421 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18333696->18337791] (req 18333696->18337791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760700.907889:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207421 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18333696->18337791] (req 18333696->18337791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760700.908026:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484132608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44740us (45183us total) trans 4294971794 rc 0/0 00000100:00100000:0.0:1789760700.908039:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4505 00000100:00100000:0.0:1789760700.908051:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690035616872 time 690194187892 state 0x2 ref 0 00000100:00100000:2.0:1789760700.913017:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760700.986416:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971795 is committed 00000100:00100000:2.0:1789760700.992529:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760701.046193:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971796 is committed 00000100:00100000:0.0:1789760701.051386:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760701.051589:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484132992 00000100:00100000:2.0:1789760701.051602:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484132992 00000100:00100000:2.0:1789760701.051680:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690244751051 time 690323755336 state 0x0 ref 1 00000100:00100000:2.0:1789760701.051698:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690324015752 time 690323755336 state 0x1 ref 1 00000100:00100000:2.0:1789760701.051929:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690324015752 time 690382857898 state 0x0 ref 1 00000100:00100000:2.0:1789760701.051946:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484132992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760701.052008:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18345984, mtime: 0x6aad94bc, atime: 0x0, ctime: 0x6aad94bc, blocks: 0x9007 00010000:00010000:2.0:1789760701.052030:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.052052:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:2.0:1789760701.052057:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760701.052084:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18345984->18350079] (req 18345984->18350079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.052103:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18345984->18350079] (req 18345984->18350079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760701.100054:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18345984 -> 18350080 00010000:00010000:0.0:1789760701.100108:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18345984->18350079] (req 18345984->18350079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.100124:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18345984->18350079] (req 18345984->18350079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.100135:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18345984->18350079] (req 18345984->18350079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.100145:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760701.100151:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18345984->18350079] (req 18345984->18350079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.100161:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18345984->18350079] (req 18345984->18350079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.100185:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18345984->18350079] (req 18345984->18350079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.100193:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207436 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18345984->18350079] (req 18345984->18350079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760701.100309:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484132992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48359us (48932us total) trans 4294971797 rc 0/0 00000100:00100000:0.0:1789760701.100322:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4508 00000100:00100000:0.0:1789760701.100335:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690324015752 time 690382857898 state 0x2 ref 0 00000100:00100000:2.0:1789760701.104634:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760701.104912:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484133120 00000100:00100000:0.0:1789760701.104927:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484133120 00000100:00100000:0.0:1789760701.105010:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690324015752 time 690382857898 state 0x0 ref 1 00000100:00100000:0.0:1789760701.106975:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690383118314 time 690382857898 state 0x1 ref 1 00000100:00100000:0.0:1789760701.107124:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690383118314 time 690438051861 state 0x0 ref 1 00000100:00100000:0.0:1789760701.107165:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484133120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760701.107236:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18350080, mtime: 0x6aad94bc, atime: 0x0, ctime: 0x6aad94bc, blocks: 0x9007 00010000:00010000:0.0:1789760701.107247:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.107268:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:0.0:1789760701.107274:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760701.107283:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18350080->18354175] (req 18350080->18354175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.107300:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18350080->18354175] (req 18350080->18354175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760701.155195:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18350080 -> 18354176 00010000:00010000:0.0:1789760701.155226:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18350080->18354175] (req 18350080->18354175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.155244:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18350080->18354175] (req 18350080->18354175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.155256:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18350080->18354175] (req 18350080->18354175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.155264:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760701.155270:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18350080->18354175] (req 18350080->18354175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.155282:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18350080->18354175] (req 18350080->18354175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.155309:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18350080->18354175] (req 18350080->18354175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.155318:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620743d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18350080->18354175] (req 18350080->18354175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760701.155473:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484133120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48324us (50852us total) trans 4294971798 rc 0/0 00000100:00100000:0.0:1789760701.155487:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4509 00000100:00100000:0.0:1789760701.155500:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690383118314 time 690438051861 state 0x2 ref 0 00000001:00080000:2.0:1789760701.160112:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971798 is committed 00000100:00100000:1.0:1789760701.161964:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484133248 00000100:00100000:1.0:1789760701.161982:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484133248 00000100:00100000:1.0:1789760701.162095:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690383118314 time 690438051861 state 0x0 ref 1 00000100:00100000:1.0:1789760701.162115:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690438312277 time 690438051861 state 0x1 ref 1 00000100:00100000:1.0:1789760701.162150:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690438312277 time 690493080699 state 0x0 ref 1 00000100:00100000:1.0:1789760701.162167:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484133248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760701.162235:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18354176, mtime: 0x6aad94bc, atime: 0x0, ctime: 0x6aad94bc, blocks: 0x9007 00010000:00010000:1.0:1789760701.162247:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.162268:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:1.0:1789760701.162273:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760701.162281:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18354176->18358271] (req 18354176->18358271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.162297:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18354176->18358271] (req 18354176->18358271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760701.213661:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971799 is committed 00002000:00010000:2.0:1789760701.213915:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18354176 -> 18358272 00002000:00010000:2.0:1789760701.213921:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760700 -> 1789760701 00002000:00010000:2.0:1789760701.213924:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760700 -> 1789760701 00010000:00010000:2.0:1789760701.213941:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18354176->18358271] (req 18354176->18358271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.213956:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18354176->18358271] (req 18354176->18358271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.213966:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18354176->18358271] (req 18354176->18358271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.213976:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760701.213982:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18354176->18358271] (req 18354176->18358271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.213998:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18354176->18358271] (req 18354176->18358271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.214166:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18354176->18358271] (req 18354176->18358271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.214179:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207444 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18354176->18358271] (req 18354176->18358271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760701.214479:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484133248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52310us (54890us total) trans 4294971799 rc 0/0 00000100:00100000:2.0:1789760701.214493:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4510 00000100:00100000:2.0:1789760701.214505:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690438312277 time 690493080699 state 0x2 ref 0 00000100:00100000:2.0:1789760701.219007:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484133376 00000100:00100000:2.0:1789760701.219029:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484133376 00000100:00100000:2.0:1789760701.219141:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690438312277 time 690493080699 state 0x0 ref 1 00000100:00100000:2.0:1789760701.219161:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690493341115 time 690493080699 state 0x1 ref 1 00000100:00100000:2.0:1789760701.219197:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690493341115 time 690550128088 state 0x0 ref 1 00000100:00100000:2.0:1789760701.219212:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484133376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760701.219425:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18358272, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:2.0:1789760701.219438:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.219459:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:2.0:1789760701.219464:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760701.219472:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18358272->18362367] (req 18358272->18362367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.219489:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18358272->18362367] (req 18358272->18362367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760701.261464:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18358272 -> 18362368 00010000:00010000:2.0:1789760701.261485:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18358272->18362367] (req 18358272->18362367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.261503:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18358272->18362367] (req 18358272->18362367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.261515:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18358272->18362367] (req 18358272->18362367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.261524:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760701.261530:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18358272->18362367] (req 18358272->18362367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.261541:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18358272->18362367] (req 18358272->18362367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.261566:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18358272->18362367] (req 18358272->18362367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.261576:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620744b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18358272->18362367] (req 18358272->18362367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760701.261854:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484133376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42637us (43038us total) trans 4294971800 rc 0/0 00000100:00100000:2.0:1789760701.261866:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4511 00000100:00100000:2.0:1789760701.261879:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690493341115 time 690550128088 state 0x2 ref 0 00000100:00100000:0.0:1789760701.265166:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760701.265332:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484133504 00000100:00100000:1.0:1789760701.265345:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484133504 00000100:00100000:1.0:1789760701.265483:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690493341115 time 690550128088 state 0x0 ref 1 00000100:00100000:1.0:1789760701.265502:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690550388504 time 690550128088 state 0x1 ref 1 00000100:00100000:1.0:1789760701.265541:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690550388504 time 690596471585 state 0x0 ref 1 00000100:00100000:1.0:1789760701.265559:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484133504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760701.265630:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18362368, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:1.0:1789760701.265641:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207452 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.265661:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:1.0:1789760701.265666:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760701.265673:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207452 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18362368->18366463] (req 18362368->18366463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.265692:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207452 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18362368->18366463] (req 18362368->18366463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760701.306533:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971801 is committed 00000100:00100000:0.0:1789760701.311252:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760701.311473:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484133632 00000100:00100000:1.0:1789760701.311486:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484133632 00000100:00100000:1.0:1789760701.311554:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690550388504 time 690596471585 state 0x0 ref 1 00000100:00100000:1.0:1789760701.311573:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690596732001 time 690596471585 state 0x1 ref 1 00000100:00100000:1.0:1789760701.311609:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690596732001 time 690642539515 state 0x0 ref 1 00000100:00100000:1.0:1789760701.311631:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484133632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760701.311695:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18366464, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:1.0:1789760701.311715:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207459 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.311841:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:1.0:1789760701.311845:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760701.311853:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207459 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18366464->18370559] (req 18366464->18370559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.311871:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207459 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18366464->18370559] (req 18366464->18370559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.326173:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760701.367392:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971802 is committed 00000100:00100000:2.0:1789760701.376296:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484133760 00000100:00100000:2.0:1789760701.376308:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484133760 00000100:00100000:2.0:1789760701.376382:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690596732001 time 690642539515 state 0x0 ref 1 00000100:00100000:2.0:1789760701.376412:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690642799931 time 690642539515 state 0x1 ref 1 00000100:00100000:2.0:1789760701.376445:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690642799931 time 690707375679 state 0x0 ref 1 00000100:00100000:2.0:1789760701.376460:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484133760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760701.376523:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18370560, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:2.0:1789760701.376534:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.376555:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:2.0:1789760701.376560:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760701.376568:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18370560->18374655] (req 18370560->18374655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.376586:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18370560->18374655] (req 18370560->18374655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760701.513392:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18370560 -> 18374656 00010000:00010000:2.0:1789760701.513425:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18370560->18374655] (req 18370560->18374655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.513441:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18370560->18374655] (req 18370560->18374655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.513452:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18370560->18374655] (req 18370560->18374655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.513461:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760701.513467:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18370560->18374655] (req 18370560->18374655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.513478:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18370560->18374655] (req 18370560->18374655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.513506:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18370560->18374655] (req 18370560->18374655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.513515:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207460 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18370560->18374655] (req 18370560->18374655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760701.513660:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484133760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 137194us (137647us total) trans 4294971803 rc 0/0 00000100:00100000:2.0:1789760701.513682:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4514 00000100:00100000:2.0:1789760701.513696:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690642799931 time 690707375679 state 0x2 ref 0 00000100:00100000:0.0:1789760701.518644:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:2.0:1789760701.564956:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18374656 -> 18378752 00010000:00010000:2.0:1789760701.564978:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207467 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18374656->18378751] (req 18374656->18378751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.564994:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207467 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18374656->18378751] (req 18374656->18378751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.565004:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207467 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18374656->18378751] (req 18374656->18378751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.565012:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760701.565019:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207467 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18374656->18378751] (req 18374656->18378751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.565030:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207467 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18374656->18378751] (req 18374656->18378751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.565089:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207467 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18374656->18378751] (req 18374656->18378751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.565104:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207467 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18374656->18378751] (req 18374656->18378751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760701.565235:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484133888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46014us (46610us total) trans 4294971804 rc 0/0 00000100:00100000:2.0:1789760701.565249:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4515 00000100:00100000:2.0:1789760701.565263:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690707636095 time 690850065993 state 0x2 ref 0 00000100:00100000:0.0:1789760701.585301:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:2.0:1789760701.617545:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18378752 -> 18382848 00010000:00010000:2.0:1789760701.617566:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620746e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18378752->18382847] (req 18378752->18382847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.617582:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620746e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18378752->18382847] (req 18378752->18382847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.617594:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620746e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18378752->18382847] (req 18378752->18382847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.617603:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760701.617609:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620746e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18378752->18382847] (req 18378752->18382847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.617632:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620746e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18378752->18382847] (req 18378752->18382847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.617663:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620746e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18378752->18382847] (req 18378752->18382847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.617673:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620746e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18378752->18382847] (req 18378752->18382847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760701.618265:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484134016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32671us (32969us total) trans 4294971805 rc 0/0 00000100:00100000:2.0:1789760701.618280:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4516 00000100:00100000:2.0:1789760701.618293:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690850326409 time 690916503373 state 0x2 ref 0 00000100:00100000:1.0:1789760701.622271:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760701.622333:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484134144 00000100:00100000:0.0:1789760701.622346:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484134144 00000100:00100000:0.0:1789760701.622414:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690850326409 time 690916503373 state 0x0 ref 1 00000100:00100000:0.0:1789760701.622428:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690916763789 time 690916503373 state 0x1 ref 1 00000100:00100000:0.0:1789760701.622457:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690916763789 time 690953389378 state 0x0 ref 1 00000100:00100000:0.0:1789760701.622468:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb727480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484134144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760701.622520:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18382848, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:0.0:1789760701.622529:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.622545:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:0.0:1789760701.622549:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760701.622555:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18382848->18386943] (req 18382848->18386943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.622569:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18382848->18386943] (req 18382848->18386943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760701.653857:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971806 is committed 00002000:00010000:2.0:1789760701.654052:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18382848 -> 18386944 00010000:00010000:2.0:1789760701.654106:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18382848->18386943] (req 18382848->18386943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.654122:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18382848->18386943] (req 18382848->18386943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.654133:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18382848->18386943] (req 18382848->18386943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.654142:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760701.654149:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18382848->18386943] (req 18382848->18386943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.654167:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18382848->18386943] (req 18382848->18386943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.654195:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18382848->18386943] (req 18382848->18386943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.654203:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207475 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18382848->18386943] (req 18382848->18386943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760701.654324:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb727480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484134144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31850us (32057us total) trans 4294971806 rc 0/0 00000100:00100000:2.0:1789760701.654336:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4517 00000100:00100000:2.0:1789760701.654347:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690916763789 time 690953389378 state 0x2 ref 0 00000100:00100000:0.0:1789760701.658422:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760701.658507:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484134272 00000100:00100000:2.0:1789760701.658521:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484134272 00000100:00100000:2.0:1789760701.658601:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690916763789 time 690953389378 state 0x0 ref 1 00000100:00100000:2.0:1789760701.658632:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690953649794 time 690953389378 state 0x1 ref 1 00000100:00100000:2.0:1789760701.658671:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690953649794 time 690989601707 state 0x0 ref 1 00000100:00100000:2.0:1789760701.658685:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484134272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760701.658754:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18386944, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:2.0:1789760701.658765:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.658789:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=690s v=(0 0) 00010000:00010000:2.0:1789760701.658794:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760701.658803:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18386944->18391039] (req 18386944->18391039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.658822:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18386944->18391039] (req 18386944->18391039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760701.670186:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760701.670271:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760701.670309:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760701.670329:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760701.670357:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760701.670390:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484134656 00000100:00100000:1.0:1789760701.670404:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484134656 00000100:00100000:1.0:1789760701.670480:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484134912 00000100:00100000:1.0:1789760701.670484:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484134912 00000100:00100000:1.0:1789760701.670505:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484134528 00000100:00100000:1.0:1789760701.670508:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484134528 00000100:00100000:1.0:1789760701.670522:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484134784 00000100:00100000:1.0:1789760701.670525:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484134784 00000100:00100000:1.0:1789760701.670543:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 841 00000100:00100000:1.0:1789760701.670553:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484134656:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760701.670662:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484134656:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 110us (390us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760701.670673:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 841 00000100:00100000:1.0:1789760701.670692:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 842 00000100:00100000:1.0:1789760701.670698:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484134912:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760701.670725:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484134912:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 28us (416us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760701.670731:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 842 00000100:00100000:1.0:1789760701.670743:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 843 00000100:00100000:1.0:1789760701.670749:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb726300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484134528:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760701.670772:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb726300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484134528:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 24us (443us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760701.670778:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 843 00000100:00100000:1.0:1789760701.670788:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 844 00000100:00080000:1.0:1789760701.671026:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:1.0:1789760701.671031:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:1.0:1789760701.671205:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484134784:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760701.671245:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484134784:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 419us (888us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760701.671252:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 844 00000100:00100000:2.0:1789760701.673211:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484134400 00000100:00100000:2.0:1789760701.673230:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484134400 00000100:00100000:2.0:1789760701.673306:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 463 00000100:00100000:2.0:1789760701.673318:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4d2910c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2425:x1876699484134400:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760701.673426:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4d2910c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2425:x1876699484134400:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 109us (3253us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760701.673433:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 463 00000001:00080000:0.0:1789760701.685821:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971807 is committed 00002000:00010000:1.0:1789760701.685935:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18386944 -> 18391040 00010000:00010000:1.0:1789760701.685956:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18386944->18391039] (req 18386944->18391039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.685971:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18386944->18391039] (req 18386944->18391039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.685987:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18386944->18391039] (req 18386944->18391039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.685995:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760701.686001:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18386944->18391039] (req 18386944->18391039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.686011:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18386944->18391039] (req 18386944->18391039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.686040:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18386944->18391039] (req 18386944->18391039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.686048:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620747c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18386944->18391039] (req 18386944->18391039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760701.686285:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484134272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27594us (27869us total) trans 4294971807 rc 0/0 00000100:00100000:1.0:1789760701.686297:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4518 00000100:00100000:1.0:1789760701.686309:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690953649794 time 690989601707 state 0x2 ref 0 00000100:00100000:1.0:1789760701.692874:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760701.692976:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484135040 00000100:00100000:2.0:1789760701.692990:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484135040 00000100:00100000:2.0:1789760701.693111:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690953649794 time 690989601707 state 0x0 ref 1 00000100:00100000:2.0:1789760701.693133:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690989862123 time 690989601707 state 0x1 ref 1 00000100:00100000:2.0:1789760701.693163:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690989862123 time 691024094103 state 0x0 ref 1 00000100:00100000:2.0:1789760701.693178:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb727100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484135040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760701.693244:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18391040, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:2.0:1789760701.693314:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.693335:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:2.0:1789760701.693340:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760701.693347:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18391040->18395135] (req 18391040->18395135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.693364:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18391040->18395135] (req 18391040->18395135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760701.724188:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971808 is committed 00002000:00010000:0.0:1789760701.724343:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18391040 -> 18395136 00010000:00010000:0.0:1789760701.724369:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18391040->18395135] (req 18391040->18395135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.724386:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18391040->18395135] (req 18391040->18395135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.724525:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18391040->18395135] (req 18391040->18395135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.724542:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760701.724548:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18391040->18395135] (req 18391040->18395135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.724559:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18391040->18395135] (req 18391040->18395135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.724585:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18391040->18395135] (req 18391040->18395135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.724595:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207483 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18391040->18395135] (req 18391040->18395135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760701.724737:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb727100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484135040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31555us (31873us total) trans 4294971808 rc 0/0 00000100:00100000:0.0:1789760701.724749:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4519 00000100:00100000:0.0:1789760701.724762:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690989862123 time 691024094103 state 0x2 ref 0 00000100:00100000:2.0:1789760701.727658:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760701.727765:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484135168 00000100:00100000:1.0:1789760701.727778:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484135168 00000100:00100000:1.0:1789760701.727851:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 690989862123 time 691024094103 state 0x0 ref 1 00000100:00100000:1.0:1789760701.727869:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691024354519 time 691024094103 state 0x1 ref 1 00000100:00100000:1.0:1789760701.727907:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691024354519 time 691058837495 state 0x0 ref 1 00000100:00100000:1.0:1789760701.727922:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484135168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760701.727988:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18395136, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:1.0:1789760701.728356:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.728380:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:1.0:1789760701.728386:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760701.728501:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18395136->18399231] (req 18395136->18399231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.728522:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18395136->18399231] (req 18395136->18399231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760701.783485:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971809 is committed 00002000:00010000:2.0:1789760701.784646:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18395136 -> 18399232 00010000:00010000:2.0:1789760701.784669:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18395136->18399231] (req 18395136->18399231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.784684:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18395136->18399231] (req 18395136->18399231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.784694:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18395136->18399231] (req 18395136->18399231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.784702:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760701.784708:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18395136->18399231] (req 18395136->18399231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.784718:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18395136->18399231] (req 18395136->18399231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.784744:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18395136->18399231] (req 18395136->18399231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.784753:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620748a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18395136->18399231] (req 18395136->18399231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760701.784869:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484135168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56942us (57217us total) trans 4294971809 rc 0/0 00000100:00100000:2.0:1789760701.784881:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4520 00000100:00100000:2.0:1789760701.784894:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691024354519 time 691058837495 state 0x2 ref 0 00000100:00100000:1.0:1789760701.788329:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760701.788499:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484135296 00000100:00100000:0.0:1789760701.788514:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484135296 00000100:00100000:0.0:1789760701.788595:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691024354519 time 691058837495 state 0x0 ref 1 00000100:00100000:0.0:1789760701.788624:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691059097911 time 691058837495 state 0x1 ref 1 00000100:00100000:0.0:1789760701.788692:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691059097911 time 691119621512 state 0x0 ref 1 00000100:00100000:0.0:1789760701.788713:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484135296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760701.788783:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18399232, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:0.0:1789760701.788794:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.788815:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:0.0:1789760701.788820:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760701.788827:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18399232->18403327] (req 18399232->18403327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.788844:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18399232->18403327] (req 18399232->18403327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760701.820229:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971810 is committed 00002000:00010000:0.0:1789760701.820282:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18399232 -> 18403328 00010000:00010000:0.0:1789760701.820303:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18399232->18403327] (req 18399232->18403327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.820315:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18399232->18403327] (req 18399232->18403327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.820322:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18399232->18403327] (req 18399232->18403327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.820327:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760701.820332:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18399232->18403327] (req 18399232->18403327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.820340:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18399232->18403327] (req 18399232->18403327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.820364:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18399232->18403327] (req 18399232->18403327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760701.820368:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207491 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18399232->18403327] (req 18399232->18403327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760701.820485:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484135296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31772us (32165us total) trans 4294971810 rc 0/0 00000100:00100000:0.0:1789760701.820495:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4521 00000100:00100000:0.0:1789760701.820506:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691059097911 time 691119621512 state 0x2 ref 0 00000100:00100000:1.0:1789760701.824485:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760701.824565:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484135424 00000100:00100000:2.0:1789760701.824578:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484135424 00000100:00100000:2.0:1789760701.824665:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691059097911 time 691119621512 state 0x0 ref 1 00000100:00100000:2.0:1789760701.824680:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691119881928 time 691119621512 state 0x1 ref 1 00000100:00100000:2.0:1789760701.824710:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691119881928 time 691155639965 state 0x0 ref 1 00000100:00100000:2.0:1789760701.824724:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484135424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760701.824791:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18403328, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:2.0:1789760701.824802:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.824822:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:2.0:1789760701.824827:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760701.824833:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18403328->18407423] (req 18403328->18407423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760701.824846:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18403328->18407423] (req 18403328->18407423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760701.869817:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18403328 -> 18407424 00010000:00010000:1.0:1789760701.869848:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18403328->18407423] (req 18403328->18407423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.869869:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18403328->18407423] (req 18403328->18407423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.869879:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18403328->18407423] (req 18403328->18407423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.869887:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760701.869893:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18403328->18407423] (req 18403328->18407423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.869903:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18403328->18407423] (req 18403328->18407423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.869928:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18403328->18407423] (req 18403328->18407423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.869937:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207498 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18403328->18407423] (req 18403328->18407423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760701.870117:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484135424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45388us (45636us total) trans 4294971811 rc 0/0 00000100:00100000:1.0:1789760701.870138:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4522 00000100:00100000:1.0:1789760701.870150:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691119881928 time 691155639965 state 0x2 ref 0 00000001:00080000:0.0:1789760701.870848:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971811 is committed 00000100:00100000:2.0:1789760701.875961:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760701.921670:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971812 is committed 00000100:00100000:1.0:1789760701.932680:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484135680 00000100:00100000:1.0:1789760701.932697:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484135680 00000100:00100000:1.0:1789760701.932778:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691155900381 time 691208024842 state 0x0 ref 1 00000100:00100000:1.0:1789760701.932799:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691208285258 time 691208024842 state 0x1 ref 1 00000100:00100000:1.0:1789760701.932894:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691208285258 time 691263823423 state 0x0 ref 1 00000100:00100000:1.0:1789760701.932910:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484135680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760701.932979:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18411520, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:1.0:1789760701.932989:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.933010:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:1.0:1789760701.933015:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760701.933023:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18411520->18415615] (req 18411520->18415615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.933042:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18411520->18415615] (req 18411520->18415615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760701.988387:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971813 is committed 00002000:00010000:1.0:1789760701.988628:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18411520 -> 18415616 00010000:00010000:1.0:1789760701.988648:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18411520->18415615] (req 18411520->18415615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.988664:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18411520->18415615] (req 18411520->18415615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.988675:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18411520->18415615] (req 18411520->18415615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.988683:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760701.988689:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18411520->18415615] (req 18411520->18415615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.988699:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18411520->18415615] (req 18411520->18415615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.988724:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18411520->18415615] (req 18411520->18415615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760701.988734:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074a6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18411520->18415615] (req 18411520->18415615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760701.988868:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484135680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55954us (57177us total) trans 4294971813 rc 0/0 00000100:00100000:1.0:1789760701.988881:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4524 00000100:00100000:1.0:1789760701.988893:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691208285258 time 691263823423 state 0x2 ref 0 00000100:00100000:1.0:1789760702.006624:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760702.006864:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484135808 00000100:00100000:0.0:1789760702.006882:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484135808 00000100:00100000:0.0:1789760702.006992:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691208285258 time 691263823423 state 0x0 ref 1 00000100:00100000:0.0:1789760702.007012:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691264083839 time 691263823423 state 0x1 ref 1 00000100:00100000:0.0:1789760702.007058:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691264083839 time 691337988878 state 0x0 ref 1 00000100:00100000:0.0:1789760702.007125:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb726a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484135808:12345-192.168.203.10@tcp:4:dd.0 00000100:00080000:1.0:1789760702.007647:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:0.0:1789760702.007916:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18415616, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:0.0:1789760702.007930:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760702.007950:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:0.0:1789760702.007955:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760702.007963:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18415616->18419711] (req 18415616->18419711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760702.007979:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18415616->18419711] (req 18415616->18419711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760702.106639:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971814 is committed 00002000:00010000:2.0:1789760702.106820:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18415616 -> 18419712 00010000:00010000:2.0:1789760702.106901:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18415616->18419711] (req 18415616->18419711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.106917:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18415616->18419711] (req 18415616->18419711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.106927:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18415616->18419711] (req 18415616->18419711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.106934:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760702.106940:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18415616->18419711] (req 18415616->18419711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.106951:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18415616->18419711] (req 18415616->18419711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.106976:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18415616->18419711] (req 18415616->18419711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.106988:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962074ad lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18415616->18419711] (req 18415616->18419711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760702.107314:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb726a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484135808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 100198us (100710us total) trans 4294971814 rc 0/0 00000100:00100000:2.0:1789760702.107329:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4525 00000100:00100000:2.0:1789760702.107342:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691264083839 time 691337988878 state 0x2 ref 0 00000100:00100000:1.0:1789760702.111720:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484135936 00000100:00100000:1.0:1789760702.111734:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484135936 00000100:00100000:1.0:1789760702.111836:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691264083839 time 691337988878 state 0x0 ref 1 00000100:00100000:1.0:1789760702.111860:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691338249294 time 691337988878 state 0x1 ref 1 00000100:00100000:1.0:1789760702.111900:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691338249294 time 691442829475 state 0x0 ref 1 00000100:00100000:1.0:1789760702.111915:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484135936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760702.111991:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18419712, mtime: 0x6aad94bd, atime: 0x0, ctime: 0x6aad94bd, blocks: 0x9007 00010000:00010000:1.0:1789760702.112004:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.112024:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:1.0:1789760702.112029:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760702.112037:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18419712->18423807] (req 18419712->18423807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.112054:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18419712->18423807] (req 18419712->18423807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760702.164658:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971815 is committed 00002000:00010000:1.0:1789760702.164925:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18419712 -> 18423808 00002000:00010000:1.0:1789760702.164932:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760701 -> 1789760702 00002000:00010000:1.0:1789760702.164935:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760701 -> 1789760702 00010000:00010000:1.0:1789760702.164952:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18419712->18423807] (req 18419712->18423807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.164966:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18419712->18423807] (req 18419712->18423807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.164977:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18419712->18423807] (req 18419712->18423807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.164986:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760702.164991:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18419712->18423807] (req 18419712->18423807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.165002:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18419712->18423807] (req 18419712->18423807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.165028:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18419712->18423807] (req 18419712->18423807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.165036:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962074b4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18419712->18423807] (req 18419712->18423807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760702.165372:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484135936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53331us (53706us total) trans 4294971815 rc 0/0 00000100:00100000:1.0:1789760702.165387:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4526 00000100:00100000:1.0:1789760702.165409:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691338249294 time 691442829475 state 0x2 ref 0 00000100:00100000:2.0:1789760702.172623:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484136064 00000100:00100000:2.0:1789760702.172642:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484136064 00000100:00100000:2.0:1789760702.172896:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691338249294 time 691442829475 state 0x0 ref 1 00000100:00100000:2.0:1789760702.172917:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691443089891 time 691442829475 state 0x1 ref 1 00000100:00100000:2.0:1789760702.172962:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691443089891 time 691503892585 state 0x0 ref 1 00000100:00100000:2.0:1789760702.172979:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484136064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760702.173047:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18423808, mtime: 0x6aad94be, atime: 0x0, ctime: 0x6aad94be, blocks: 0x9007 00010000:00010000:2.0:1789760702.173059:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962074bb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.173110:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:2.0:1789760702.173116:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760702.173123:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962074bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18423808->18427903] (req 18423808->18427903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.173146:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962074bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18423808->18427903] (req 18423808->18427903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760702.255207:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971816 is committed 00000100:00100000:0.0:1789760702.260921:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760702.348269:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501ba0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508903552:0@lo:13:osp-pre-0-1.0 00000100:00100000:2.0:1789760702.348372:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760702.348482:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508903552 00000100:00100000:2.0:1789760702.348493:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508903552 00000100:00100000:2.0:1789760702.348577:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 350 00000100:00100000:2.0:1789760702.348588:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508903552:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:2.0:1789760702.348631:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528154624 pending 1703936 free 3839885312 avail 3828350976 00000100:00100000:2.0:1789760702.348687:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508903552:12345-0@lo:13:osp-pre-0-1.0 Request processed in 100us (315us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760702.348707:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 350 00000100:00080000:2.0:1789760702.348735:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760702.348778:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501ba0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508903552:0@lo:13:osp-pre-0-1.0 00000001:00080000:0.0:1789760702.355192:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971817 is committed 00000100:00100000:0.0:1789760702.355731:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508903680 00000100:00100000:0.0:1789760702.355742:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508903680 00000100:00100000:0.0:1789760702.355951:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 409 00000100:00100000:0.0:1789760702.355968:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508903680:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:0.0:1789760702.356025:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:2.0:1789760702.357713:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760702.368227:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508903680:12345-0@lo:41:osp-pre-0-1.0 Request processed in 12250us (12791us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760702.368259:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 409 00000001:00080000:1.0:1789760702.413579:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971818 is committed 00002000:00010000:0.0:1789760702.413730:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18432000 -> 18436096 00010000:00010000:0.0:1789760702.413751:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962074c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18432000->18436095] (req 18432000->18436095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760702.413767:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962074c9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18432000->18436095] (req 18432000->18436095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760702.413778:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962074c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18432000->18436095] (req 18432000->18436095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760702.413786:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760702.413792:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962074c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18432000->18436095] (req 18432000->18436095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760702.413804:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962074c9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18432000->18436095] (req 18432000->18436095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760702.413832:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962074c9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18432000->18436095] (req 18432000->18436095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760702.413841:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962074c9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18432000->18436095] (req 18432000->18436095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760702.413992:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484136320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56039us (56292us total) trans 4294971818 rc 0/0 00000100:00100000:0.0:1789760702.414005:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4529 00000100:00100000:0.0:1789760702.414019:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691592876284 time 691688865067 state 0x2 ref 0 00000100:00100000:1.0:1789760702.426768:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484136448 00000100:00100000:1.0:1789760702.426784:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484136448 00000100:00100000:1.0:1789760702.426862:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691592876284 time 691688865067 state 0x0 ref 1 00000100:00100000:1.0:1789760702.426886:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691689125483 time 691688865067 state 0x1 ref 1 00000100:00100000:1.0:1789760702.426921:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691689125483 time 691757851761 state 0x0 ref 1 00000100:00100000:1.0:1789760702.426937:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484136448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760702.427001:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18436096, mtime: 0x6aad94be, atime: 0x0, ctime: 0x6aad94be, blocks: 0x9007 00010000:00010000:1.0:1789760702.427013:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962074d0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.427033:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:1.0:1789760702.427039:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760702.427047:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962074d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18436096->18440191] (req 18436096->18440191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.427121:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962074d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18436096->18440191] (req 18436096->18440191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760702.479319:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971819 is committed 00000100:00100000:1.0:1789760702.486461:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760702.486786:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484136576 00000100:00100000:2.0:1789760702.486800:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484136576 00000100:00100000:2.0:1789760702.486881:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691689125483 time 691757851761 state 0x0 ref 1 00000100:00100000:2.0:1789760702.486900:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691758112177 time 691757851761 state 0x1 ref 1 00000100:00100000:2.0:1789760702.486936:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691758112177 time 691817866337 state 0x0 ref 1 00000100:00100000:2.0:1789760702.486951:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb727b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484136576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760702.487022:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18440192, mtime: 0x6aad94be, atime: 0x0, ctime: 0x6aad94be, blocks: 0x9007 00010000:00010000:2.0:1789760702.487033:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.487055:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=691s v=(0 0) 00010000:00010000:2.0:1789760702.487061:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760702.487324:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18440192->18444287] (req 18440192->18444287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.487344:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18440192->18444287] (req 18440192->18444287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760702.533343:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971820 is committed 00002000:00010000:2.0:1789760702.533553:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18440192 -> 18444288 00010000:00010000:2.0:1789760702.533576:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18440192->18444287] (req 18440192->18444287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.533593:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18440192->18444287] (req 18440192->18444287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.533605:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18440192->18444287] (req 18440192->18444287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.533622:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760702.533627:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18440192->18444287] (req 18440192->18444287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.533637:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18440192->18444287] (req 18440192->18444287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.533665:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18440192->18444287] (req 18440192->18444287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.533674:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962074d7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18440192->18444287] (req 18440192->18444287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760702.533927:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb727b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484136576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46971us (47476us total) trans 4294971820 rc 0/0 00000100:00100000:2.0:1789760702.533940:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4531 00000100:00100000:2.0:1789760702.533953:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691758112177 time 691817866337 state 0x2 ref 0 00000100:00100000:2.0:1789760702.537366:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760702.612285:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971821 is committed 00000100:00100000:0.0:1789760702.617986:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760702.668590:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971822 is committed 00002000:00010000:1.0:1789760702.671181:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18448384 -> 18452480 00010000:00010000:1.0:1789760702.671211:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962074e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18448384->18452479] (req 18448384->18452479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.671228:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962074e5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18448384->18452479] (req 18448384->18452479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.671245:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962074e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18448384->18452479] (req 18448384->18452479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.671254:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760702.671261:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962074e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18448384->18452479] (req 18448384->18452479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.671271:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962074e5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18448384->18452479] (req 18448384->18452479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.671302:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962074e5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18448384->18452479] (req 18448384->18452479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760702.671310:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962074e5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18448384->18452479] (req 18448384->18452479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760702.671459:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484136832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53100us (53480us total) trans 4294971822 rc 0/0 00000100:00100000:1.0:1789760702.671471:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4533 00000100:00100000:1.0:1789760702.671484:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 691869374516 time 691949268840 state 0x2 ref 0 00000100:00100000:1.0:1789760702.678454:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760702.710665:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971823 is committed 00000100:00100000:1.0:1789760702.727447:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760702.830243:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971824 is committed 00002000:00010000:2.0:1789760702.830351:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18456576 -> 18460672 00010000:00010000:2.0:1789760702.830369:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962074f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18456576->18460671] (req 18456576->18460671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.830382:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962074f3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18456576->18460671] (req 18456576->18460671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.830435:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962074f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18456576->18460671] (req 18456576->18460671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.830441:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760702.830446:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962074f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18456576->18460671] (req 18456576->18460671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.830455:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962074f3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18456576->18460671] (req 18456576->18460671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.830477:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962074f3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18456576->18460671] (req 18456576->18460671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.830481:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962074f3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18456576->18460671] (req 18456576->18460671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760702.830582:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484137088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 102835us (103149us total) trans 4294971824 rc 0/0 00000100:00100000:2.0:1789760702.830591:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4535 00000100:00100000:2.0:1789760702.830600:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692009966204 time 692058661031 state 0x2 ref 0 00000100:00100000:0.0:1789760702.833514:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760702.868557:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:1.0:1789760702.868588:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c886d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508903936:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760702.868700:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000001:00080000:1.0:1789760702.902480:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971825 is committed 00000100:00100000:1.0:1789760702.908922:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00002000:00010000:2.0:1789760702.957580:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18464768 -> 18468864 00010000:00010000:2.0:1789760702.957600:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207501 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18464768->18468863] (req 18464768->18468863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.957627:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207501 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18464768->18468863] (req 18464768->18468863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.957639:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207501 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18464768->18468863] (req 18464768->18468863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.957648:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760702.957654:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207501 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18464768->18468863] (req 18464768->18468863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.957665:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207501 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18464768->18468863] (req 18464768->18468863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.957761:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207501 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18464768->18468863] (req 18464768->18468863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760702.957771:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207501 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18464768->18468863] (req 18464768->18468863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760702.958619:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb727100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484137344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48960us (49688us total) trans 4294971826 rc 0/0 00000100:00100000:2.0:1789760702.958636:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4537 00000100:00100000:2.0:1789760702.958649:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692165074197 time 692240552954 state 0x2 ref 0 00000100:00100000:1.0:1789760702.962676:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760703.017911:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971827 is committed 00000100:00100000:0.0:1789760703.021524:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:1.0:1789760703.075208:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971828 is committed 00000100:00100000:1.0:1789760703.078824:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760703.079060:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484137728 00000100:00100000:0.0:1789760703.079118:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484137728 00000100:00100000:0.0:1789760703.079200:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692294166766 time 692352906639 state 0x0 ref 1 00000100:00100000:0.0:1789760703.079219:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692353167055 time 692352906639 state 0x1 ref 1 00000100:00100000:0.0:1789760703.079250:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692353167055 time 692410180650 state 0x0 ref 1 00000100:00100000:0.0:1789760703.079265:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484137728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760703.079332:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18477056, mtime: 0x6aad94be, atime: 0x0, ctime: 0x6aad94be, blocks: 0x9007 00010000:00010000:0.0:1789760703.079343:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760703.079365:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=692s v=(0 0) 00010000:00010000:0.0:1789760703.079370:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760703.079378:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18477056->18481151] (req 18477056->18481151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760703.079561:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18477056->18481151] (req 18477056->18481151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760703.122904:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971829 is committed 00002000:00010000:2.0:1789760703.123861:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18477056 -> 18481152 00010000:00010000:2.0:1789760703.123884:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18477056->18481151] (req 18477056->18481151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.123899:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18477056->18481151] (req 18477056->18481151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.123909:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18477056->18481151] (req 18477056->18481151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.123923:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760703.123930:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18477056->18481151] (req 18477056->18481151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.123940:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18477056->18481151] (req 18477056->18481151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.123964:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18477056->18481151] (req 18477056->18481151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.123972:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207516 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18477056->18481151] (req 18477056->18481151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760703.124148:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484137728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44879us (45329us total) trans 4294971829 rc 0/0 00000100:00100000:2.0:1789760703.124161:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4540 00000100:00100000:2.0:1789760703.124172:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692353167055 time 692410180650 state 0x2 ref 0 00000100:00100000:2.0:1789760703.131566:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760703.131847:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484137856 00000100:00100000:1.0:1789760703.131862:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484137856 00000100:00100000:1.0:1789760703.133090:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692353167055 time 692410180650 state 0x0 ref 1 00000100:00100000:1.0:1789760703.133114:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692410441066 time 692410180650 state 0x1 ref 1 00000100:00100000:1.0:1789760703.133162:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692410441066 time 692464091462 state 0x0 ref 1 00000100:00100000:1.0:1789760703.134289:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484137856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760703.134373:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18481152, mtime: 0x6aad94be, atime: 0x0, ctime: 0x6aad94be, blocks: 0x9007 00010000:00010000:1.0:1789760703.134386:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.134407:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=692s v=(0 0) 00010000:00010000:1.0:1789760703.134412:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760703.134419:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18481152->18485247] (req 18481152->18485247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.134437:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18481152->18485247] (req 18481152->18485247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760703.190578:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971830 is committed 00002000:00010000:2.0:1789760703.190733:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18481152 -> 18485248 00002000:00010000:2.0:1789760703.190739:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760702 -> 1789760703 00002000:00010000:2.0:1789760703.190741:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760702 -> 1789760703 00010000:00010000:2.0:1789760703.190759:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18481152->18485247] (req 18481152->18485247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.190790:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18481152->18485247] (req 18481152->18485247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.190800:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18481152->18485247] (req 18481152->18485247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.190809:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760703.190814:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18481152->18485247] (req 18481152->18485247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.190824:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18481152->18485247] (req 18481152->18485247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.190848:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18481152->18485247] (req 18481152->18485247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.190858:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620751d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18481152->18485247] (req 18481152->18485247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760703.190980:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484137856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57800us (59424us total) trans 4294971830 rc 0/0 00000100:00100000:2.0:1789760703.190992:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4541 00000100:00100000:2.0:1789760703.191003:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692410441066 time 692464091462 state 0x2 ref 0 00000100:00100000:0.0:1789760703.202289:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760703.281177:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971831 is committed 00000100:00100000:1.0:1789760703.291803:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760703.353109:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971832 is committed 00000100:00100000:2.0:1789760703.365949:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760703.366376:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484138240 00000100:00100000:0.0:1789760703.366490:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484138240 00000100:00100000:1.0:1789760703.366663:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692534193552 time 692623195837 state 0x0 ref 1 00000100:00100000:1.0:1789760703.366683:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692623456253 time 692623195837 state 0x1 ref 1 00000100:00100000:1.0:1789760703.366765:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692623456253 time 692697694147 state 0x0 ref 1 00000100:00100000:1.0:1789760703.366780:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484138240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760703.366846:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18493440, mtime: 0x6aad94bf, atime: 0x0, ctime: 0x6aad94bf, blocks: 0x9007 00010000:00010000:1.0:1789760703.366857:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.366877:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=692s v=(0 0) 00010000:00010000:1.0:1789760703.366882:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760703.366890:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18493440->18497535] (req 18493440->18497535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.366906:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18493440->18497535] (req 18493440->18497535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.374793:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760703.435315:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971833 is committed 00002000:00010000:1.0:1789760703.435526:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18493440 -> 18497536 00010000:00010000:1.0:1789760703.435547:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18493440->18497535] (req 18493440->18497535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.435569:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18493440->18497535] (req 18493440->18497535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.435580:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18493440->18497535] (req 18493440->18497535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.436584:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760703.436609:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18493440->18497535] (req 18493440->18497535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.436627:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18493440->18497535] (req 18493440->18497535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.436655:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18493440->18497535] (req 18493440->18497535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.436665:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207532 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18493440->18497535] (req 18493440->18497535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760703.436795:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484138240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 70010us (70853us total) trans 4294971833 rc 0/0 00000100:00100000:1.0:1789760703.436809:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4544 00000100:00100000:1.0:1789760703.436823:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692623456253 time 692697694147 state 0x2 ref 0 00000100:00100000:0.0:1789760703.441841:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760703.451256:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484138368 00000100:00100000:2.0:1789760703.451279:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484138368 00000100:00100000:2.0:1789760703.451626:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692623456253 time 692697694147 state 0x0 ref 1 00000100:00100000:2.0:1789760703.451648:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692697954563 time 692697694147 state 0x1 ref 1 00000100:00100000:2.0:1789760703.451690:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692697954563 time 692782620076 state 0x0 ref 1 00000100:00100000:2.0:1789760703.451707:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484138368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760703.451780:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18497536, mtime: 0x6aad94bf, atime: 0x0, ctime: 0x6aad94bf, blocks: 0x9007 00010000:00010000:2.0:1789760703.451791:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.451811:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=692s v=(0 0) 00010000:00010000:2.0:1789760703.451817:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760703.451825:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18497536->18501631] (req 18497536->18501631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.451843:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18497536->18501631] (req 18497536->18501631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760703.539852:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971834 is committed 00002000:00010000:2.0:1789760703.540148:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18497536 -> 18501632 00010000:00010000:2.0:1789760703.540168:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18497536->18501631] (req 18497536->18501631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.540184:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18497536->18501631] (req 18497536->18501631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.540195:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18497536->18501631] (req 18497536->18501631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.540203:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760703.540208:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18497536->18501631] (req 18497536->18501631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.540217:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18497536->18501631] (req 18497536->18501631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.540241:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18497536->18501631] (req 18497536->18501631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.540249:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207539 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18497536->18501631] (req 18497536->18501631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760703.540371:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484138368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 88661us (98539us total) trans 4294971834 rc 0/0 00000100:00100000:2.0:1789760703.540390:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4545 00000100:00100000:2.0:1789760703.540403:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692697954563 time 692782620076 state 0x2 ref 0 00000100:00100000:0.0:1789760703.545989:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760703.646456:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971835 is committed 00002000:00010000:1.0:1789760703.646758:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18501632 -> 18505728 00010000:00010000:1.0:1789760703.646779:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207540 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18501632->18505727] (req 18501632->18505727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.646795:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207540 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18501632->18505727] (req 18501632->18505727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.646804:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207540 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18501632->18505727] (req 18501632->18505727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.646812:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760703.646818:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207540 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18501632->18505727] (req 18501632->18505727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.646829:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207540 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18501632->18505727] (req 18501632->18505727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.646856:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207540 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18501632->18505727] (req 18501632->18505727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.646865:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207540 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18501632->18505727] (req 18501632->18505727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760703.646985:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484138496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 100723us (101002us total) trans 4294971835 rc 0/0 00000100:00100000:1.0:1789760703.646998:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4546 00000100:00100000:1.0:1789760703.647010:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692782880492 time 692877174235 state 0x2 ref 0 00000100:00100000:1.0:1789760703.654628:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760703.654793:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484138624 00000100:00100000:0.0:1789760703.654806:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484138624 00000100:00100000:0.0:1789760703.654887:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692782880492 time 692877174235 state 0x0 ref 1 00000100:00100000:0.0:1789760703.654907:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692877434651 time 692877174235 state 0x1 ref 1 00000100:00100000:0.0:1789760703.654943:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692877434651 time 692985874080 state 0x0 ref 1 00000100:00100000:0.0:1789760703.654958:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484138624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760703.655023:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18505728, mtime: 0x6aad94bf, atime: 0x0, ctime: 0x6aad94bf, blocks: 0x9007 00010000:00010000:0.0:1789760703.655035:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760703.655053:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=692s v=(0 0) 00010000:00010000:0.0:1789760703.655058:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760703.655105:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18505728->18509823] (req 18505728->18509823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760703.655123:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18505728->18509823] (req 18505728->18509823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760703.728010:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971836 is committed 00002000:00010000:2.0:1789760703.731951:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18505728 -> 18509824 00010000:00010000:2.0:1789760703.731982:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18505728->18509823] (req 18505728->18509823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.732006:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18505728->18509823] (req 18505728->18509823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.732018:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18505728->18509823] (req 18505728->18509823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.732026:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760703.732031:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18505728->18509823] (req 18505728->18509823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.732042:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18505728->18509823] (req 18505728->18509823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.732096:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18505728->18509823] (req 18505728->18509823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.732105:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207547 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18505728->18509823] (req 18505728->18509823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760703.732245:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484138624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77281us (77623us total) trans 4294971836 rc 0/0 00000100:00100000:2.0:1789760703.732258:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4547 00000100:00100000:2.0:1789760703.732270:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692877434651 time 692985874080 state 0x2 ref 0 00000100:00100000:0.0:1789760703.736028:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760703.736230:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484138752 00000100:00100000:2.0:1789760703.736245:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484138752 00000100:00100000:2.0:1789760703.736317:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692877434651 time 692985874080 state 0x0 ref 1 00000100:00100000:2.0:1789760703.736336:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692986134496 time 692985874080 state 0x1 ref 1 00000100:00100000:2.0:1789760703.736372:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692986134496 time 693067302034 state 0x0 ref 1 00000100:00100000:2.0:1789760703.736395:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484138752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760703.736466:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18509824, mtime: 0x6aad94bf, atime: 0x0, ctime: 0x6aad94bf, blocks: 0x9007 00010000:00010000:2.0:1789760703.736478:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.736496:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:2.0:1789760703.736502:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760703.736509:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18509824->18513919] (req 18509824->18513919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.736526:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18509824->18513919] (req 18509824->18513919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760703.853972:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18509824 -> 18513920 00010000:00010000:1.0:1789760703.854000:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18509824->18513919] (req 18509824->18513919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.854015:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18509824->18513919] (req 18509824->18513919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.854026:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18509824->18513919] (req 18509824->18513919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.854034:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760703.854039:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18509824->18513919] (req 18509824->18513919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.854049:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18509824->18513919] (req 18509824->18513919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.854115:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18509824->18513919] (req 18509824->18513919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760703.854123:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620754e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18509824->18513919] (req 18509824->18513919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760703.854236:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484138752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 117839us (118215us total) trans 4294971837 rc 0/0 00000100:00100000:1.0:1789760703.854249:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4548 00000100:00100000:1.0:1789760703.855677:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692986134496 time 693067302034 state 0x2 ref 0 00000100:00100000:1.0:1789760703.858366:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:0.0:1789760703.858527:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971837 is committed 00000100:00100000:2.0:1789760703.858648:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484138880 00000100:00100000:2.0:1789760703.858661:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484138880 00000100:00100000:2.0:1789760703.858789:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 692986134496 time 693067302034 state 0x0 ref 1 00000100:00100000:2.0:1789760703.858807:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693067562450 time 693067302034 state 0x1 ref 1 00000100:00100000:2.0:1789760703.858876:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693067562450 time 693189805950 state 0x0 ref 1 00000100:00100000:2.0:1789760703.858893:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484138880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760703.858958:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18513920, mtime: 0x6aad94bf, atime: 0x0, ctime: 0x6aad94bf, blocks: 0x9007 00010000:00010000:2.0:1789760703.858969:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.858990:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:2.0:1789760703.858995:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760703.859003:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18513920->18518015] (req 18513920->18518015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.859019:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18513920->18518015] (req 18513920->18518015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760703.928054:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18513920 -> 18518016 00010000:00010000:2.0:1789760703.928102:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18513920->18518015] (req 18513920->18518015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.928119:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18513920->18518015] (req 18513920->18518015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.928128:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18513920->18518015] (req 18513920->18518015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.928137:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760703.928142:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18513920->18518015] (req 18513920->18518015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.928154:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18513920->18518015] (req 18513920->18518015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.928180:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18513920->18518015] (req 18513920->18518015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.928189:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207555 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18513920->18518015] (req 18513920->18518015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760703.928325:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484138880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69427us (69965us total) trans 4294971838 rc 0/0 00000100:00100000:2.0:1789760703.928338:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4549 00000100:00100000:2.0:1789760703.928350:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693067562450 time 693189805950 state 0x2 ref 0 00000100:00100000:2.0:1789760703.931964:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000001:00080000:2.0:1789760703.984044:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971839 is committed 00000100:00100000:3.0:1789760703.984846:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484139008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52396us (52891us total) trans 4294971839 rc 0/0 00000100:00100000:3.0:1789760703.984863:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4550 00000100:00100000:3.0:1789760703.984876:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693190066366 time 693263359125 state 0x2 ref 0 00000100:00100000:0.0:1789760703.989456:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760703.989550:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484139136 00000100:00100000:2.0:1789760703.989563:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484139136 00000100:00100000:2.0:1789760703.989644:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693190066366 time 693263359125 state 0x0 ref 1 00000100:00100000:2.0:1789760703.989662:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693263619541 time 693263359125 state 0x1 ref 1 00000100:00100000:2.0:1789760703.989814:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693263619541 time 693320742927 state 0x0 ref 1 00000100:00100000:2.0:1789760703.989830:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484139136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760703.989900:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18522112, mtime: 0x6aad94bf, atime: 0x0, ctime: 0x6aad94bf, blocks: 0x9007 00010000:00010000:2.0:1789760703.989912:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.989933:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:2.0:1789760703.989938:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760703.989945:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18522112->18526207] (req 18522112->18526207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760703.989963:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18522112->18526207] (req 18522112->18526207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760704.052022:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971840 is committed 00002000:00010000:2.0:1789760704.052180:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18522112 -> 18526208 00010000:00010000:2.0:1789760704.052200:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18522112->18526207] (req 18522112->18526207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.052216:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18522112->18526207] (req 18522112->18526207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.052227:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18522112->18526207] (req 18522112->18526207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.052235:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760704.052241:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18522112->18526207] (req 18522112->18526207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.052251:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18522112->18526207] (req 18522112->18526207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.052278:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18522112->18526207] (req 18522112->18526207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.052287:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207563 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18522112->18526207] (req 18522112->18526207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760704.052415:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484139136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62581us (62968us total) trans 4294971840 rc 0/0 00000100:00100000:2.0:1789760704.052427:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4551 00000100:00100000:2.0:1789760704.052439:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693263619541 time 693320742927 state 0x2 ref 0 00000100:00100000:1.0:1789760704.057831:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760704.057932:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484139264 00000100:00100000:2.0:1789760704.057945:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484139264 00000100:00100000:2.0:1789760704.058129:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693263619541 time 693320742927 state 0x0 ref 1 00000100:00100000:2.0:1789760704.058147:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693321003343 time 693320742927 state 0x1 ref 1 00000100:00100000:2.0:1789760704.058185:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693321003343 time 693389115215 state 0x0 ref 1 00000100:00100000:2.0:1789760704.058201:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484139264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760704.058268:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18526208, mtime: 0x6aad94bf, atime: 0x0, ctime: 0x6aad94bf, blocks: 0x9007 00010000:00010000:2.0:1789760704.058280:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.058300:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:2.0:1789760704.058306:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760704.058314:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18526208->18530303] (req 18526208->18530303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.058331:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18526208->18530303] (req 18526208->18530303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760704.063871:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:1.0:1789760704.149720:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971841 is committed 00002000:00010000:2.0:1789760704.149771:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18526208 -> 18530304 00010000:00010000:2.0:1789760704.149796:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18526208->18530303] (req 18526208->18530303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.149813:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18526208->18530303] (req 18526208->18530303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.149825:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18526208->18530303] (req 18526208->18530303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.149833:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760704.149839:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18526208->18530303] (req 18526208->18530303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.149848:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18526208->18530303] (req 18526208->18530303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.149874:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18526208->18530303] (req 18526208->18530303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.149882:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620756a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18526208->18530303] (req 18526208->18530303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760704.150611:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484139264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 92393us (92859us total) trans 4294971841 rc 0/0 00000100:00100000:2.0:1789760704.150627:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4552 00000100:00100000:2.0:1789760704.150640:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693321003343 time 693389115215 state 0x2 ref 0 00000100:00100000:1.0:1789760704.156525:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760704.156650:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484139392 00000100:00100000:2.0:1789760704.156663:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484139392 00000100:00100000:2.0:1789760704.156759:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693321003343 time 693389115215 state 0x0 ref 1 00000100:00100000:2.0:1789760704.156778:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693389375631 time 693389115215 state 0x1 ref 1 00000100:00100000:2.0:1789760704.156814:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693389375631 time 693487744526 state 0x0 ref 1 00000100:00100000:2.0:1789760704.156828:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484139392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760704.156898:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18530304, mtime: 0x6aad94bf, atime: 0x0, ctime: 0x6aad94bf, blocks: 0x9007 00010000:00010000:2.0:1789760704.156919:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.156939:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:2.0:1789760704.156944:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760704.156952:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18530304->18534399] (req 18530304->18534399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.156969:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18530304->18534399] (req 18530304->18534399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760704.217993:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971842 is committed 00002000:00010000:2.0:1789760704.218231:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18530304 -> 18534400 00002000:00010000:2.0:1789760704.218237:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760703 -> 1789760704 00002000:00010000:2.0:1789760704.218240:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760703 -> 1789760704 00010000:00010000:2.0:1789760704.218256:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18530304->18534399] (req 18530304->18534399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.218271:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18530304->18534399] (req 18530304->18534399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.218282:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18530304->18534399] (req 18530304->18534399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.218292:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760704.218299:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18530304->18534399] (req 18530304->18534399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.218310:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18530304->18534399] (req 18530304->18534399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.218335:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18530304->18534399] (req 18530304->18534399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.218345:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207571 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18530304->18534399] (req 18530304->18534399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760704.218546:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484139392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61713us (62027us total) trans 4294971842 rc 0/0 00000100:00100000:2.0:1789760704.219402:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4553 00000100:00100000:2.0:1789760704.219422:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693389375631 time 693487744526 state 0x2 ref 0 00000100:00100000:3.0:1789760704.221977:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760704.222188:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484139520 00000100:00100000:1.0:1789760704.222201:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484139520 00000100:00100000:1.0:1789760704.222275:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693389375631 time 693487744526 state 0x0 ref 1 00000100:00100000:1.0:1789760704.222294:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693488004942 time 693487744526 state 0x1 ref 1 00000100:00100000:1.0:1789760704.222323:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693488004942 time 693553254111 state 0x0 ref 1 00000100:00100000:1.0:1789760704.222339:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484139520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760704.222418:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18534400, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:1.0:1789760704.222429:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760704.222451:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:1.0:1789760704.222456:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760704.222465:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18534400->18538495] (req 18534400->18538495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760704.222488:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18534400->18538495] (req 18534400->18538495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760704.259936:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971843 is committed 00002000:00010000:1.0:1789760704.260105:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18534400 -> 18538496 00010000:00010000:1.0:1789760704.260128:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18534400->18538495] (req 18534400->18538495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760704.260145:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18534400->18538495] (req 18534400->18538495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760704.260157:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18534400->18538495] (req 18534400->18538495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760704.260166:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760704.260172:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18534400->18538495] (req 18534400->18538495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760704.260182:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18534400->18538495] (req 18534400->18538495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760704.260207:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18534400->18538495] (req 18534400->18538495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760704.260216:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207578 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18534400->18538495] (req 18534400->18538495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760704.260339:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484139520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37995us (38367us total) trans 4294971843 rc 0/0 00000100:00100000:1.0:1789760704.260352:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4554 00000100:00100000:1.0:1789760704.260365:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693488004942 time 693553254111 state 0x2 ref 0 00000100:00100000:1.0:1789760704.264230:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760704.264437:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484139648 00000100:00100000:0.0:1789760704.264451:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484139648 00000100:00100000:0.0:1789760704.264534:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693488004942 time 693553254111 state 0x0 ref 1 00000100:00100000:0.0:1789760704.264553:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693553514527 time 693553254111 state 0x1 ref 1 00000100:00100000:0.0:1789760704.264642:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693553514527 time 693595571951 state 0x0 ref 1 00000100:00100000:0.0:1789760704.264658:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484139648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760704.264734:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18538496, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:0.0:1789760704.264746:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.264767:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:0.0:1789760704.264772:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760704.264780:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18538496->18542591] (req 18538496->18542591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.264797:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18538496->18542591] (req 18538496->18542591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760704.295605:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971844 is committed 00002000:00010000:2.0:1789760704.295743:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18538496 -> 18542592 00010000:00010000:2.0:1789760704.295764:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18538496->18542591] (req 18538496->18542591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.295781:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18538496->18542591] (req 18538496->18542591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.295792:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18538496->18542591] (req 18538496->18542591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.295799:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760704.295805:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18538496->18542591] (req 18538496->18542591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.295815:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18538496->18542591] (req 18538496->18542591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.295841:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18538496->18542591] (req 18538496->18542591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.295855:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620757f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18538496->18542591] (req 18538496->18542591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760704.295987:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484139648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31326us (31764us total) trans 4294971844 rc 0/0 00000100:00100000:2.0:1789760704.296000:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4555 00000100:00100000:2.0:1789760704.296013:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693553514527 time 693595571951 state 0x2 ref 0 00000100:00100000:1.0:1789760704.300249:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760704.300347:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484139776 00000100:00100000:3.0:1789760704.300361:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484139776 00000100:00100000:3.0:1789760704.300495:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693553514527 time 693595571951 state 0x0 ref 1 00000100:00100000:3.0:1789760704.300516:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693595832367 time 693595571951 state 0x1 ref 1 00000100:00100000:3.0:1789760704.300551:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693595832367 time 693631481779 state 0x0 ref 1 00000100:00100000:3.0:1789760704.300566:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484139776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760704.300641:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18542592, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:3.0:1789760704.300653:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.300674:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:3.0:1789760704.300788:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760704.300798:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18542592->18546687] (req 18542592->18546687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.300817:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18542592->18546687] (req 18542592->18546687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760704.336433:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18542592 -> 18546688 00010000:00010000:0.0:1789760704.336458:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18542592->18546687] (req 18542592->18546687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.336473:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18542592->18546687] (req 18542592->18546687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.336490:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18542592->18546687] (req 18542592->18546687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.336498:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760704.336503:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18542592->18546687] (req 18542592->18546687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.336513:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18542592->18546687] (req 18542592->18546687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.336535:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18542592->18546687] (req 18542592->18546687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.336542:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207586 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18542592->18546687] (req 18542592->18546687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760704.339366:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484139776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36111us (36440us total) trans 4294971845 rc 0/0 00000100:00100000:0.0:1789760704.339409:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4556 00000100:00100000:0.0:1789760704.339424:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693595832367 time 693631481779 state 0x2 ref 0 00000001:00080000:3.0:1789760704.339715:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971845 is committed 00000100:00100000:0.0:1789760704.341356:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760704.341546:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484139904 00000100:00100000:2.0:1789760704.341642:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484139904 00000100:00100000:2.0:1789760704.341885:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693595832367 time 693631481779 state 0x0 ref 1 00000100:00100000:2.0:1789760704.341903:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693631742195 time 693631481779 state 0x1 ref 1 00000100:00100000:2.0:1789760704.342014:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693631742195 time 693672942957 state 0x0 ref 1 00000100:00100000:2.0:1789760704.342030:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484139904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760704.342117:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18546688, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:2.0:1789760704.342128:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.342150:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:2.0:1789760704.342155:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760704.342232:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18546688->18550783] (req 18546688->18550783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.342259:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18546688->18550783] (req 18546688->18550783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760704.376882:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971846 is committed 00002000:00010000:2.0:1789760704.377144:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18546688 -> 18550784 00010000:00010000:2.0:1789760704.377174:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18546688->18550783] (req 18546688->18550783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.377191:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18546688->18550783] (req 18546688->18550783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.377202:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18546688->18550783] (req 18546688->18550783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.377211:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760704.377217:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18546688->18550783] (req 18546688->18550783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.377228:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18546688->18550783] (req 18546688->18550783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.377252:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18546688->18550783] (req 18546688->18550783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.377261:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620758d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18546688->18550783] (req 18546688->18550783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760704.377510:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484139904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35475us (36158us total) trans 4294971846 rc 0/0 00000100:00100000:2.0:1789760704.377523:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4557 00000100:00100000:2.0:1789760704.377538:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693631742195 time 693672942957 state 0x2 ref 0 00000100:00100000:1.0:1789760704.383964:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760704.384108:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484140032 00000100:00100000:0.0:1789760704.384121:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484140032 00000100:00100000:0.0:1789760704.384215:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693631742195 time 693672942957 state 0x0 ref 1 00000100:00100000:0.0:1789760704.384239:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693673203373 time 693672942957 state 0x1 ref 1 00000100:00100000:0.0:1789760704.384275:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693673203373 time 693715205554 state 0x0 ref 1 00000100:00100000:0.0:1789760704.384290:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484140032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760704.384354:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18550784, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:0.0:1789760704.384365:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.384397:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:0.0:1789760704.384402:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760704.384411:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18550784->18554879] (req 18550784->18554879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.384430:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18550784->18554879] (req 18550784->18554879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.396495:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760704.435371:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971847 is committed 00002000:00010000:3.0:1789760704.435553:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18550784 -> 18554880 00010000:00010000:3.0:1789760704.435572:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18550784->18554879] (req 18550784->18554879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.435588:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18550784->18554879] (req 18550784->18554879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.435609:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18550784->18554879] (req 18550784->18554879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.435617:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760704.435623:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18550784->18554879] (req 18550784->18554879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.435633:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18550784->18554879] (req 18550784->18554879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.435660:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18550784->18554879] (req 18550784->18554879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.435668:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207594 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18550784->18554879] (req 18550784->18554879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760704.436173:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484140032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51878us (52214us total) trans 4294971847 rc 0/0 00000100:00100000:3.0:1789760704.436186:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4558 00000100:00100000:3.0:1789760704.436199:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693673203373 time 693715205554 state 0x2 ref 0 00000100:00100000:1.0:1789760704.442197:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760704.442291:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484140160 00000100:00100000:2.0:1789760704.442305:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484140160 00000100:00100000:2.0:1789760704.442387:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693673203373 time 693715205554 state 0x0 ref 1 00000100:00100000:2.0:1789760704.442407:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693715465970 time 693715205554 state 0x1 ref 1 00000100:00100000:2.0:1789760704.442440:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693715465970 time 693773370465 state 0x0 ref 1 00000100:00100000:2.0:1789760704.442454:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484140160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760704.442519:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18554880, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:2.0:1789760704.442529:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.442549:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:2.0:1789760704.442555:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760704.442562:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18554880->18558975] (req 18554880->18558975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.442579:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18554880->18558975] (req 18554880->18558975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760704.510493:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971848 is committed 00002000:00010000:3.0:1789760704.510952:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18554880 -> 18558976 00010000:00010000:3.0:1789760704.510973:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18554880->18558975] (req 18554880->18558975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.510988:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18554880->18558975] (req 18554880->18558975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.510998:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18554880->18558975] (req 18554880->18558975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.511006:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760704.511012:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18554880->18558975] (req 18554880->18558975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.511021:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18554880->18558975] (req 18554880->18558975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.511046:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18554880->18558975] (req 18554880->18558975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.511060:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620759b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18554880->18558975] (req 18554880->18558975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760704.511209:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484140160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68751us (69018us total) trans 4294971848 rc 0/0 00000100:00100000:3.0:1789760704.511222:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4559 00000100:00100000:3.0:1789760704.511234:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693715465970 time 693773370465 state 0x2 ref 0 00000100:00100000:3.0:1789760704.516163:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760704.516269:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484140288 00000100:00100000:0.0:1789760704.516282:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484140288 00000100:00100000:0.0:1789760704.516363:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693715465970 time 693773370465 state 0x0 ref 1 00000100:00100000:0.0:1789760704.516392:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693773630881 time 693773370465 state 0x1 ref 1 00000100:00100000:0.0:1789760704.516428:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693773630881 time 693847358441 state 0x0 ref 1 00000100:00100000:0.0:1789760704.516448:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484140288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760704.516511:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18558976, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:0.0:1789760704.516522:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.516542:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:0.0:1789760704.516548:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760704.516556:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18558976->18563071] (req 18558976->18563071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.516575:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18558976->18563071] (req 18558976->18563071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760704.602607:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971849 is committed 00002000:00010000:3.0:1789760704.602954:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18558976 -> 18563072 00010000:00010000:3.0:1789760704.602977:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18558976->18563071] (req 18558976->18563071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.602991:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18558976->18563071] (req 18558976->18563071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.603001:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18558976->18563071] (req 18558976->18563071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.603009:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760704.603014:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18558976->18563071] (req 18558976->18563071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.603023:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18558976->18563071] (req 18558976->18563071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.603052:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18558976->18563071] (req 18558976->18563071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.603060:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962075a2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18558976->18563071] (req 18558976->18563071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760704.603228:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484140288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 86776us (87073us total) trans 4294971849 rc 0/0 00000100:00100000:3.0:1789760704.603241:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4560 00000100:00100000:3.0:1789760704.603254:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693773630881 time 693847358441 state 0x2 ref 0 00000100:00100000:1.0:1789760704.608200:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760704.608345:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484140416 00000100:00100000:2.0:1789760704.608359:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484140416 00000100:00100000:2.0:1789760704.608446:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693773630881 time 693847358441 state 0x0 ref 1 00000100:00100000:2.0:1789760704.608466:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693847618857 time 693847358441 state 0x1 ref 1 00000100:00100000:2.0:1789760704.608497:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693847618857 time 693939427920 state 0x0 ref 1 00000100:00100000:2.0:1789760704.608512:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484140416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760704.608577:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18563072, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:2.0:1789760704.608588:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.608618:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=693s v=(0 0) 00010000:00010000:2.0:1789760704.608623:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760704.608631:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18563072->18567167] (req 18563072->18567167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.608647:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18563072->18567167] (req 18563072->18567167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760704.669286:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971850 is committed 00002000:00010000:3.0:1789760704.669499:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18563072 -> 18567168 00010000:00010000:3.0:1789760704.669519:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18563072->18567167] (req 18563072->18567167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.669535:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18563072->18567167] (req 18563072->18567167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.669545:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18563072->18567167] (req 18563072->18567167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.669553:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760704.669558:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18563072->18567167] (req 18563072->18567167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.669574:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18563072->18567167] (req 18563072->18567167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.669614:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18563072->18567167] (req 18563072->18567167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.669624:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962075a9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18563072->18567167] (req 18563072->18567167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760704.669898:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484140416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61381us (61704us total) trans 4294971850 rc 0/0 00000100:00100000:3.0:1789760704.669910:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4561 00000100:00100000:3.0:1789760704.669923:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693847618857 time 693939427920 state 0x2 ref 0 00000100:00100000:1.0:1789760704.673440:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760704.673521:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484140544 00000100:00100000:3.0:1789760704.673534:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484140544 00000100:00100000:3.0:1789760704.673615:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693847618857 time 693939427920 state 0x0 ref 1 00000100:00100000:3.0:1789760704.673635:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693939688336 time 693939427920 state 0x1 ref 1 00000100:00100000:3.0:1789760704.673671:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693939688336 time 694004601495 state 0x0 ref 1 00000100:00100000:3.0:1789760704.673749:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484140544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760704.673887:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18567168, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:3.0:1789760704.673899:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.673922:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:3.0:1789760704.673927:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760704.673937:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18567168->18571263] (req 18567168->18571263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.673959:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18567168->18571263] (req 18567168->18571263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760704.753230:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18567168 -> 18571264 00010000:00010000:3.0:1789760704.753260:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18567168->18571263] (req 18567168->18571263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.753276:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18567168->18571263] (req 18567168->18571263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.753286:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18567168->18571263] (req 18567168->18571263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.753294:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760704.753300:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18567168->18571263] (req 18567168->18571263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.753310:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18567168->18571263] (req 18567168->18571263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.753334:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18567168->18571263] (req 18567168->18571263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.753344:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075b0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18567168->18571263] (req 18567168->18571263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760704.753543:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484140544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79790us (80180us total) trans 4294971851 rc 0/0 00000100:00100000:3.0:1789760704.753563:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4562 00000100:00100000:3.0:1789760704.753578:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693939688336 time 694004601495 state 0x2 ref 0 00000001:00080000:1.0:1789760704.753744:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971851 is committed 00000100:00100000:0.0:1789760704.756880:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760704.757326:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484140672 00000100:00100000:3.0:1789760704.757340:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484140672 00000100:00100000:3.0:1789760704.757901:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 693939688336 time 694004601495 state 0x0 ref 1 00000100:00100000:3.0:1789760704.757924:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694004861911 time 694004601495 state 0x1 ref 1 00000100:00100000:3.0:1789760704.758033:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694004861911 time 694088962129 state 0x0 ref 1 00000100:00100000:3.0:1789760704.758054:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484140672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760704.758177:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18571264, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:3.0:1789760704.758190:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.758211:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:3.0:1789760704.758216:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760704.758223:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18571264->18575359] (req 18571264->18575359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.758246:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18571264->18575359] (req 18571264->18575359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760704.859734:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971852 is committed 00002000:00010000:3.0:1789760704.861872:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18571264 -> 18575360 00010000:00010000:3.0:1789760704.861900:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18571264->18575359] (req 18571264->18575359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.861916:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18571264->18575359] (req 18571264->18575359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.861926:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18571264->18575359] (req 18571264->18575359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.861934:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760704.861940:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18571264->18575359] (req 18571264->18575359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.861950:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18571264->18575359] (req 18571264->18575359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.861978:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18571264->18575359] (req 18571264->18575359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.861986:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075b7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18571264->18575359] (req 18571264->18575359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760704.862165:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484140672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 104112us (105292us total) trans 4294971852 rc 0/0 00000100:00100000:3.0:1789760704.862178:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4563 00000100:00100000:3.0:1789760704.862191:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694004861911 time 694088962129 state 0x2 ref 0 00000100:00100000:0.0:1789760704.868744:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760704.869033:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484140800 00000100:00100000:3.0:1789760704.869047:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484140800 00000100:00100000:3.0:1789760704.871775:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694004861911 time 694088962129 state 0x0 ref 1 00000100:00100000:3.0:1789760704.871792:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694089222545 time 694088962129 state 0x1 ref 1 00000100:00100000:3.0:1789760704.871833:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694089222545 time 694202765039 state 0x0 ref 1 00000100:00100000:3.0:1789760704.871846:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484140800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760704.871893:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18575360, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:3.0:1789760704.871900:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.871977:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:3.0:1789760704.871982:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760704.871989:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18575360->18579455] (req 18575360->18579455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.872003:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18575360->18579455] (req 18575360->18579455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760704.906295:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18575360 -> 18579456 00010000:00010000:2.0:1789760704.906314:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18575360->18579455] (req 18575360->18579455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.906332:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18575360->18579455] (req 18575360->18579455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.906340:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18575360->18579455] (req 18575360->18579455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.906346:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760704.906351:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18575360->18579455] (req 18575360->18579455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.906360:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18575360->18579455] (req 18575360->18579455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.906380:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18575360->18579455] (req 18575360->18579455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.906387:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962075be lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18575360->18579455] (req 18575360->18579455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760704.906494:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971853 is committed 00000100:00100000:2.0:1789760704.906547:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484140800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34630us (37806us total) trans 4294971853 rc 0/0 00000100:00100000:2.0:1789760704.906560:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4564 00000100:00100000:2.0:1789760704.906569:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694089222545 time 694202765039 state 0x2 ref 0 00000100:00100000:1.0:1789760704.911890:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760704.912303:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484140928 00000100:00100000:2.0:1789760704.912314:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484140928 00000100:00100000:2.0:1789760704.912369:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694089222545 time 694202765039 state 0x0 ref 1 00000100:00100000:2.0:1789760704.912384:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694203025455 time 694202765039 state 0x1 ref 1 00000100:00100000:2.0:1789760704.912580:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694203025455 time 694243509082 state 0x0 ref 1 00000100:00100000:2.0:1789760704.912605:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484140928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760704.912670:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18579456, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:2.0:1789760704.912680:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.912696:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:2.0:1789760704.912700:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760704.912706:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18579456->18583551] (req 18579456->18583551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760704.912720:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18579456->18583551] (req 18579456->18583551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760704.966169:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971854 is committed 00002000:00010000:0.0:1789760704.966449:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18579456 -> 18583552 00010000:00010000:0.0:1789760704.966464:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18579456->18583551] (req 18579456->18583551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.966477:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18579456->18583551] (req 18579456->18583551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.966557:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18579456->18583551] (req 18579456->18583551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.966570:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760704.966576:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18579456->18583551] (req 18579456->18583551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.966585:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18579456->18583551] (req 18579456->18583551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.966616:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18579456->18583551] (req 18579456->18583551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760704.966624:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075c5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18579456->18583551] (req 18579456->18583551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760704.966815:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484140928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54208us (54928us total) trans 4294971854 rc 0/0 00000100:00100000:0.0:1789760704.966826:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4565 00000100:00100000:0.0:1789760704.966835:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694203025455 time 694243509082 state 0x2 ref 0 00000100:00100000:0.0:1789760704.970466:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760704.970622:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484141056 00000100:00100000:3.0:1789760704.970736:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484141056 00000100:00100000:3.0:1789760704.970792:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694203025455 time 694243509082 state 0x0 ref 1 00000100:00100000:3.0:1789760704.970807:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694243769498 time 694243509082 state 0x1 ref 1 00000100:00100000:3.0:1789760704.970886:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694243769498 time 694301815946 state 0x0 ref 1 00000100:00100000:3.0:1789760704.970898:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484141056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760704.970944:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18583552, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:3.0:1789760704.970953:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.970970:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:3.0:1789760704.970974:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760704.970980:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18583552->18587647] (req 18583552->18587647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.970995:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18583552->18587647] (req 18583552->18587647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760704.995428:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971855 is committed 00002000:00010000:3.0:1789760704.995559:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18583552 -> 18587648 00010000:00010000:3.0:1789760704.995579:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18583552->18587647] (req 18583552->18587647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.995609:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18583552->18587647] (req 18583552->18587647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.995618:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18583552->18587647] (req 18583552->18587647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.995629:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760704.995634:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18583552->18587647] (req 18583552->18587647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.995643:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18583552->18587647] (req 18583552->18587647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.995662:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18583552->18587647] (req 18583552->18587647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760704.995669:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962075cc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18583552->18587647] (req 18583552->18587647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760704.996048:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484141056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25146us (25586us total) trans 4294971855 rc 0/0 00000100:00100000:3.0:1789760704.996059:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4566 00000100:00100000:3.0:1789760704.997401:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694243769498 time 694301815946 state 0x2 ref 0 00000100:00100000:1.0:1789760705.001859:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760705.001933:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484141184 00000100:00100000:3.0:1789760705.001943:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484141184 00000100:00100000:3.0:1789760705.001997:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694243769498 time 694301815946 state 0x0 ref 1 00000100:00100000:3.0:1789760705.002012:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694302076362 time 694301815946 state 0x1 ref 1 00000100:00100000:3.0:1789760705.002138:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694302076362 time 694333066644 state 0x0 ref 1 00000100:00100000:3.0:1789760705.002151:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484141184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760705.002198:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18587648, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:3.0:1789760705.002205:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.002218:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:3.0:1789760705.002221:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760705.002227:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18587648->18591743] (req 18587648->18591743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.002241:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18587648->18591743] (req 18587648->18591743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760705.034025:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18587648 -> 18591744 00010000:00010000:3.0:1789760705.034046:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18587648->18591743] (req 18587648->18591743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.034060:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18587648->18591743] (req 18587648->18591743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.034107:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18587648->18591743] (req 18587648->18591743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.034114:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760705.034119:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18587648->18591743] (req 18587648->18591743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.034127:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18587648->18591743] (req 18587648->18591743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.034146:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18587648->18591743] (req 18587648->18591743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.034153:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075d3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18587648->18591743] (req 18587648->18591743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760705.034235:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484141184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32081us (32381us total) trans 4294971856 rc 0/0 00000100:00100000:3.0:1789760705.034245:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4567 00000100:00100000:3.0:1789760705.034253:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694302076362 time 694333066644 state 0x2 ref 0 00000001:00080000:1.0:1789760705.034905:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971856 is committed 00000100:00100000:1.0:1789760705.040247:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760705.040350:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484141312 00000100:00100000:2.0:1789760705.040360:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484141312 00000100:00100000:2.0:1789760705.040412:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694302076362 time 694333066644 state 0x0 ref 1 00000100:00100000:2.0:1789760705.040426:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694333327060 time 694333066644 state 0x1 ref 1 00000100:00100000:2.0:1789760705.040454:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694333327060 time 694371385494 state 0x0 ref 1 00000100:00100000:2.0:1789760705.040469:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484141312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760705.040517:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18591744, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:2.0:1789760705.040526:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.040541:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:2.0:1789760705.040546:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760705.040553:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18591744->18595839] (req 18591744->18595839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.040567:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18591744->18595839] (req 18591744->18595839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760705.124570:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971857 is committed 00002000:00010000:2.0:1789760705.124798:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18591744 -> 18595840 00010000:00010000:2.0:1789760705.124813:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18591744->18595839] (req 18591744->18595839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.124824:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18591744->18595839] (req 18591744->18595839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.124832:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18591744->18595839] (req 18591744->18595839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.124838:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760705.124842:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18591744->18595839] (req 18591744->18595839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.124851:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18591744->18595839] (req 18591744->18595839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.124870:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18591744->18595839] (req 18591744->18595839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.124878:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962075da lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18591744->18595839] (req 18591744->18595839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760705.125187:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484141312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 84715us (84943us total) trans 4294971857 rc 0/0 00000100:00100000:2.0:1789760705.125199:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4568 00000100:00100000:2.0:1789760705.125208:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694333327060 time 694371385494 state 0x2 ref 0 00000100:00100000:0.0:1789760705.146665:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760705.146786:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484141440 00000100:00100000:2.0:1789760705.146796:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484141440 00000100:00100000:2.0:1789760705.146854:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694333327060 time 694371385494 state 0x0 ref 1 00000100:00100000:2.0:1789760705.146882:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694371645910 time 694371385494 state 0x1 ref 1 00000100:00100000:2.0:1789760705.146912:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694371645910 time 694477842541 state 0x0 ref 1 00000100:00100000:2.0:1789760705.146925:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484141440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760705.152258:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18595840, mtime: 0x6aad94c0, atime: 0x0, ctime: 0x6aad94c0, blocks: 0x9007 00010000:00010000:0.0:1789760705.152273:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.152290:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:0.0:1789760705.152296:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760705.152303:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18595840->18599935] (req 18595840->18599935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.152318:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18595840->18599935] (req 18595840->18599935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760705.289774:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971858 is committed 00002000:00010000:0.0:1789760705.293805:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18595840 -> 18599936 00002000:00010000:0.0:1789760705.293814:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760704 -> 1789760705 00002000:00010000:0.0:1789760705.293817:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760704 -> 1789760705 00010000:00010000:0.0:1789760705.293829:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18595840->18599935] (req 18595840->18599935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.293842:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18595840->18599935] (req 18595840->18599935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.293850:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18595840->18599935] (req 18595840->18599935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.293864:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760705.293869:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18595840->18599935] (req 18595840->18599935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.293878:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18595840->18599935] (req 18595840->18599935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.293898:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18595840->18599935] (req 18595840->18599935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.293906:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962075e1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18595840->18599935] (req 18595840->18599935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760705.294820:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484141440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 147891us (148231us total) trans 4294971858 rc 0/0 00000100:00100000:0.0:1789760705.294832:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4569 00000100:00100000:0.0:1789760705.294842:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694371645910 time 694477842541 state 0x2 ref 0 00000100:00100000:1.0:1789760705.319824:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760705.324569:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484141568 00000100:00100000:3.0:1789760705.324583:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484141568 00000100:00100000:2.0:1789760705.326835:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694371645910 time 694477842541 state 0x0 ref 1 00000100:00100000:2.0:1789760705.327115:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694478102957 time 694477842541 state 0x1 ref 1 00000100:00100000:2.0:1789760705.327210:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694478102957 time 694658139425 state 0x0 ref 1 00000100:00100000:2.0:1789760705.327227:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484141568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760705.327280:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18599936, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:2.0:1789760705.327289:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.327305:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:2.0:1789760705.327310:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760705.327316:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18599936->18604031] (req 18599936->18604031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.327331:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18599936->18604031] (req 18599936->18604031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.420993:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:2.0:1789760705.439121:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18599936 -> 18604032 00000001:00080000:1.0:1789760705.439133:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971859 is committed 00010000:00010000:2.0:1789760705.439140:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18599936->18604031] (req 18599936->18604031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.439151:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18599936->18604031] (req 18599936->18604031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.439159:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18599936->18604031] (req 18599936->18604031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.439173:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760705.439178:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18599936->18604031] (req 18599936->18604031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.439186:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18599936->18604031] (req 18599936->18604031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.439206:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18599936->18604031] (req 18599936->18604031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.439213:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075e8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18599936->18604031] (req 18599936->18604031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760705.439310:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484141568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 112082us (119491us total) trans 4294971859 rc 0/0 00000100:00100000:2.0:1789760705.439320:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4570 00000100:00100000:2.0:1789760705.439328:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694478102957 time 694658139425 state 0x2 ref 0 00000100:00100000:2.0:1789760705.450276:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760705.450345:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484141696 00000100:00100000:3.0:1789760705.450355:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484141696 00000100:00100000:3.0:1789760705.450426:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694478102957 time 694658139425 state 0x0 ref 1 00000100:00100000:3.0:1789760705.450445:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694658399841 time 694658139425 state 0x1 ref 1 00000100:00100000:3.0:1789760705.450474:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694658399841 time 694781405778 state 0x0 ref 1 00000100:00100000:3.0:1789760705.450557:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484141696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760705.450620:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18604032, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:3.0:1789760705.450629:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.450646:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:3.0:1789760705.450651:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760705.450659:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18604032->18608127] (req 18604032->18608127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.450681:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18604032->18608127] (req 18604032->18608127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760705.555395:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971860 is committed 00002000:00010000:1.0:1789760705.565172:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18604032 -> 18608128 00010000:00010000:1.0:1789760705.565193:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18604032->18608127] (req 18604032->18608127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.565205:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18604032->18608127] (req 18604032->18608127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.565213:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18604032->18608127] (req 18604032->18608127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.565221:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760705.565226:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18604032->18608127] (req 18604032->18608127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.565235:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18604032->18608127] (req 18604032->18608127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.565255:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18604032->18608127] (req 18604032->18608127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.565263:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962075ef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18604032->18608127] (req 18604032->18608127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760705.568800:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484141696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 118234us (118522us total) trans 4294971860 rc 0/0 00000100:00100000:1.0:1789760705.568817:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4571 00000100:00100000:1.0:1789760705.568827:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694658399841 time 694781405778 state 0x2 ref 0 00000100:00100000:0.0:1789760705.583009:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760705.583105:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484141824 00000100:00100000:2.0:1789760705.583115:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484141824 00000100:00100000:2.0:1789760705.583166:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694658399841 time 694781405778 state 0x0 ref 1 00000100:00100000:2.0:1789760705.583182:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694781666194 time 694781405778 state 0x1 ref 1 00000100:00100000:2.0:1789760705.583211:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694781666194 time 694914142694 state 0x0 ref 1 00000100:00100000:2.0:1789760705.583225:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484141824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760705.583273:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18608128, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:2.0:1789760705.583282:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.583297:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:2.0:1789760705.583301:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760705.583308:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18608128->18612223] (req 18608128->18612223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.583322:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18608128->18612223] (req 18608128->18612223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760705.651446:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18608128 -> 18612224 00000001:00080000:2.0:1789760705.651455:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971861 is committed 00010000:00010000:0.0:1789760705.651468:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18608128->18612223] (req 18608128->18612223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.651481:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18608128->18612223] (req 18608128->18612223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.651491:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18608128->18612223] (req 18608128->18612223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.651499:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760705.651504:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18608128->18612223] (req 18608128->18612223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.651513:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18608128->18612223] (req 18608128->18612223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.651538:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18608128->18612223] (req 18608128->18612223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.651546:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962075f6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18608128->18612223] (req 18608128->18612223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760705.651693:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484141824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68462us (68685us total) trans 4294971861 rc 0/0 00000100:00100000:0.0:1789760705.651704:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4572 00000100:00100000:0.0:1789760705.651714:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694781666194 time 694914142694 state 0x2 ref 0 00000100:00100000:3.0:1789760705.655959:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760705.656397:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484141952 00000100:00100000:0.0:1789760705.656407:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484141952 00000100:00100000:0.0:1789760705.656477:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694781666194 time 694914142694 state 0x0 ref 1 00000100:00100000:0.0:1789760705.656493:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694914403110 time 694914142694 state 0x1 ref 1 00000100:00100000:0.0:1789760705.656526:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694914403110 time 694987454716 state 0x0 ref 1 00000100:00100000:1.0:1789760705.656595:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484141952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760705.656647:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18612224, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:1.0:1789760705.656656:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.656670:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=694s v=(0 0) 00010000:00010000:1.0:1789760705.656674:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760705.656680:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18612224->18616319] (req 18612224->18616319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.656694:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18612224->18616319] (req 18612224->18616319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760705.683727:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971862 is committed 00002000:00010000:0.0:1789760705.683825:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18612224 -> 18616320 00010000:00010000:0.0:1789760705.683842:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18612224->18616319] (req 18612224->18616319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.683856:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18612224->18616319] (req 18612224->18616319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.683872:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18612224->18616319] (req 18612224->18616319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.683881:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760705.683886:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18612224->18616319] (req 18612224->18616319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.683896:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18612224->18616319] (req 18612224->18616319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.683919:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18612224->18616319] (req 18612224->18616319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.683927:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962075fd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18612224->18616319] (req 18612224->18616319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760705.684023:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484141952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27480us (28064us total) trans 4294971862 rc 0/0 00000100:00100000:0.0:1789760705.684034:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4573 00000100:00100000:0.0:1789760705.684044:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694914403110 time 694987454716 state 0x2 ref 0 00000100:00100000:2.0:1789760705.688168:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760705.688844:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484142080 00000100:00100000:0.0:1789760705.688853:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484142080 00000100:00100000:0.0:1789760705.688912:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694914403110 time 694987454716 state 0x0 ref 1 00000100:00100000:0.0:1789760705.688927:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694987715132 time 694987454716 state 0x1 ref 1 00000100:00100000:0.0:1789760705.688961:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694987715132 time 695019891442 state 0x0 ref 1 00000100:00100000:0.0:1789760705.688976:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484142080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760705.689029:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18616320, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:0.0:1789760705.689037:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.689053:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:0.0:1789760705.689058:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760705.689088:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18616320->18620415] (req 18616320->18620415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.689102:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18616320->18620415] (req 18616320->18620415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760705.712709:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971863 is committed 00002000:00010000:3.0:1789760705.713000:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18616320 -> 18620416 00010000:00010000:3.0:1789760705.713013:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18616320->18620415] (req 18616320->18620415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.713023:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18616320->18620415] (req 18616320->18620415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.713029:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18616320->18620415] (req 18616320->18620415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.713035:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760705.713038:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18616320->18620415] (req 18616320->18620415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.713044:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18616320->18620415] (req 18616320->18620415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.713058:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18616320->18620415] (req 18616320->18620415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760705.713062:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207604 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18616320->18620415] (req 18616320->18620415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760705.713283:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484142080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24307us (25120us total) trans 4294971863 rc 0/0 00000100:00100000:3.0:1789760705.713294:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4574 00000100:00100000:3.0:1789760705.713303:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694987715132 time 695019891442 state 0x2 ref 0 00000100:00100000:3.0:1789760705.717249:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760705.717312:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484142208 00000100:00100000:1.0:1789760705.717320:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484142208 00000100:00100000:1.0:1789760705.717364:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 694987715132 time 695019891442 state 0x0 ref 1 00000100:00100000:1.0:1789760705.717378:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695020151858 time 695019891442 state 0x1 ref 1 00000100:00100000:1.0:1789760705.717408:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695020151858 time 695048338959 state 0x0 ref 1 00000100:00100000:1.0:1789760705.717424:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484142208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760705.717592:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18620416, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:1.0:1789760705.717603:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.717619:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:1.0:1789760705.717622:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760705.717627:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18620416->18624511] (req 18620416->18624511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.717638:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18620416->18624511] (req 18620416->18624511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760705.741635:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18620416 -> 18624512 00010000:00010000:0.0:1789760705.741653:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18620416->18624511] (req 18620416->18624511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.741666:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18620416->18624511] (req 18620416->18624511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.741675:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18620416->18624511] (req 18620416->18624511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.741683:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760705.741689:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18620416->18624511] (req 18620416->18624511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.741697:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18620416->18624511] (req 18620416->18624511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.741715:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18620416->18624511] (req 18620416->18624511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.741723:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620760b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18620416->18624511] (req 18620416->18624511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760705.741817:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484142208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24390us (24572us total) trans 4294971864 rc 0/0 00000100:00100000:0.0:1789760705.741828:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4575 00000100:00100000:0.0:1789760705.741838:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695020151858 time 695048338959 state 0x2 ref 0 00000001:00080000:1.0:1789760705.741913:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971864 is committed 00000100:00100000:0.0:1789760705.745941:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760705.746389:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484142336 00000100:00100000:2.0:1789760705.746399:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484142336 00000100:00100000:2.0:1789760705.746446:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695020151858 time 695048338959 state 0x0 ref 1 00000100:00100000:2.0:1789760705.746457:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695048599375 time 695048338959 state 0x1 ref 1 00000100:00100000:2.0:1789760705.746475:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695048599375 time 695077407432 state 0x0 ref 1 00000100:00100000:2.0:1789760705.746483:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484142336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760705.746524:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18624512, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:2.0:1789760705.746531:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.746544:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:2.0:1789760705.746547:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760705.746552:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18624512->18628607] (req 18624512->18628607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.746562:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18624512->18628607] (req 18624512->18628607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760705.770020:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971865 is committed 00002000:00010000:2.0:1789760705.770120:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18624512 -> 18628608 00010000:00010000:2.0:1789760705.770136:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18624512->18628607] (req 18624512->18628607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.770150:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18624512->18628607] (req 18624512->18628607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.770159:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18624512->18628607] (req 18624512->18628607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.770164:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760705.770167:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18624512->18628607] (req 18624512->18628607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.770173:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18624512->18628607] (req 18624512->18628607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.770196:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18624512->18628607] (req 18624512->18628607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760705.770204:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207612 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18624512->18628607] (req 18624512->18628607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760705.770426:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484142336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23937us (24489us total) trans 4294971865 rc 0/0 00000100:00100000:2.0:1789760705.770437:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4576 00000100:00100000:2.0:1789760705.770445:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695048599375 time 695077407432 state 0x2 ref 0 00000100:00100000:2.0:1789760705.772277:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760705.772339:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484142464 00000100:00100000:1.0:1789760705.772347:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484142464 00000100:00100000:1.0:1789760705.772432:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695048599375 time 695077407432 state 0x0 ref 1 00000100:00100000:1.0:1789760705.772443:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695077667848 time 695077407432 state 0x1 ref 1 00000100:00100000:1.0:1789760705.772460:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695077667848 time 695103392677 state 0x0 ref 1 00000100:00100000:1.0:1789760705.772469:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484142464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760705.772508:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18628608, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:1.0:1789760705.772514:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.772526:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:1.0:1789760705.772529:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760705.772533:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18628608->18632703] (req 18628608->18632703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.772542:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18628608->18632703] (req 18628608->18632703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760705.805263:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18628608 -> 18632704 00010000:00010000:0.0:1789760705.805283:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18628608->18632703] (req 18628608->18632703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.805294:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18628608->18632703] (req 18628608->18632703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.805303:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18628608->18632703] (req 18628608->18632703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.805315:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760705.805319:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18628608->18632703] (req 18628608->18632703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.805328:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18628608->18632703] (req 18628608->18632703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.805346:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18628608->18632703] (req 18628608->18632703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.805354:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207619 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18628608->18632703] (req 18628608->18632703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760705.805520:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484142464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33046us (33245us total) trans 4294971866 rc 0/0 00000100:00100000:0.0:1789760705.805530:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4577 00000100:00100000:0.0:1789760705.805539:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695077667848 time 695103392677 state 0x2 ref 0 00000001:00080000:3.0:1789760705.805619:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971866 is committed 00000100:00100000:3.0:1789760705.808485:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760705.808536:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484142592 00000100:00100000:0.0:1789760705.808547:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484142592 00000100:00100000:0.0:1789760705.808617:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695077667848 time 695103392677 state 0x0 ref 1 00000100:00100000:0.0:1789760705.808632:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695103653093 time 695103392677 state 0x1 ref 1 00000100:00100000:0.0:1789760705.808663:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695103653093 time 695139593350 state 0x0 ref 1 00000100:00100000:0.0:1789760705.808680:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484142592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760705.808726:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18632704, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:0.0:1789760705.808734:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.808750:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:0.0:1789760705.808754:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760705.808760:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18632704->18636799] (req 18632704->18636799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.808774:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18632704->18636799] (req 18632704->18636799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760705.834861:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971867 is committed 00002000:00010000:0.0:1789760705.834868:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18632704 -> 18636800 00010000:00010000:0.0:1789760705.834883:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18632704->18636799] (req 18632704->18636799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.834897:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18632704->18636799] (req 18632704->18636799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.834907:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18632704->18636799] (req 18632704->18636799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.834915:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760705.834920:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18632704->18636799] (req 18632704->18636799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.834930:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18632704->18636799] (req 18632704->18636799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.834949:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18632704->18636799] (req 18632704->18636799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.834958:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207620 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18632704->18636799] (req 18632704->18636799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760705.835045:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484142592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26364us (26564us total) trans 4294971867 rc 0/0 00000100:00100000:0.0:1789760705.835054:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4578 00000100:00100000:0.0:1789760705.835104:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695103653093 time 695139593350 state 0x2 ref 0 00000100:00100000:2.0:1789760705.839827:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760705.839901:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484142720 00000100:00100000:0.0:1789760705.839910:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484142720 00000100:00100000:0.0:1789760705.839962:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695103653093 time 695139593350 state 0x0 ref 1 00000100:00100000:0.0:1789760705.839977:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695139853766 time 695139593350 state 0x1 ref 1 00000100:00100000:1.0:1789760705.840250:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695139853766 time 695171178051 state 0x0 ref 1 00000100:00100000:1.0:1789760705.840267:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484142720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760705.840316:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18636800, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:1.0:1789760705.840325:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.840341:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:1.0:1789760705.840345:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760705.840352:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18636800->18640895] (req 18636800->18640895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.840368:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18636800->18640895] (req 18636800->18640895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760705.871856:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971868 is committed 00002000:00010000:0.0:1789760705.872057:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18636800 -> 18640896 00010000:00010000:0.0:1789760705.872132:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18636800->18640895] (req 18636800->18640895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.872144:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18636800->18640895] (req 18636800->18640895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.872157:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18636800->18640895] (req 18636800->18640895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.872163:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760705.872168:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18636800->18640895] (req 18636800->18640895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.872176:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18636800->18640895] (req 18636800->18640895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.872194:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18636800->18640895] (req 18636800->18640895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.872202:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207627 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18636800->18640895] (req 18636800->18640895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760705.872321:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484142720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32054us (32498us total) trans 4294971868 rc 0/0 00000100:00100000:0.0:1789760705.872331:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4579 00000100:00100000:0.0:1789760705.872340:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695139853766 time 695171178051 state 0x2 ref 0 00000100:00100000:3.0:1789760705.875963:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760705.876029:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484142848 00000100:00100000:0.0:1789760705.876039:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484142848 00000100:00100000:0.0:1789760705.876118:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695139853766 time 695171178051 state 0x0 ref 1 00000100:00100000:0.0:1789760705.876133:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695171438467 time 695171178051 state 0x1 ref 1 00000100:00100000:0.0:1789760705.876155:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695171438467 time 695207087368 state 0x0 ref 1 00000100:00100000:0.0:1789760705.876168:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484142848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760705.876212:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18640896, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:0.0:1789760705.876220:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.876236:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:0.0:1789760705.876239:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760705.876246:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18640896->18644991] (req 18640896->18644991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.876259:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18640896->18644991] (req 18640896->18644991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760705.904202:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971869 is committed 00002000:00010000:0.0:1789760705.904322:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18640896 -> 18644992 00010000:00010000:0.0:1789760705.904337:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18640896->18644991] (req 18640896->18644991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.904350:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18640896->18644991] (req 18640896->18644991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.904358:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18640896->18644991] (req 18640896->18644991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.904435:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760705.904440:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18640896->18644991] (req 18640896->18644991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.904450:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18640896->18644991] (req 18640896->18644991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.904469:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18640896->18644991] (req 18640896->18644991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.904477:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620762e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18640896->18644991] (req 18640896->18644991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760705.904613:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484142848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28443us (28654us total) trans 4294971869 rc 0/0 00000100:00100000:0.0:1789760705.904623:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4580 00000100:00100000:0.0:1789760705.904632:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695171438467 time 695207087368 state 0x2 ref 0 00000100:00100000:2.0:1789760705.907630:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760705.907776:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484142976 00000100:00100000:0.0:1789760705.907786:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484142976 00000100:00100000:0.0:1789760705.907845:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695171438467 time 695207087368 state 0x0 ref 1 00000100:00100000:0.0:1789760705.907860:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695207347784 time 695207087368 state 0x1 ref 1 00000100:00100000:0.0:1789760705.907933:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695207347784 time 695238863526 state 0x0 ref 1 00000100:00100000:0.0:1789760705.907947:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484142976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760705.907993:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18644992, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:0.0:1789760705.908002:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.908017:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:0.0:1789760705.908021:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760705.908027:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18644992->18649087] (req 18644992->18649087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.908041:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18644992->18649087] (req 18644992->18649087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760705.932202:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4defdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508904064:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760705.932269:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760705.932332:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508904064 00000100:00100000:2.0:1789760705.932340:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508904064 00000100:00100000:2.0:1789760705.932397:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 352 00000100:00100000:2.0:1789760705.932405:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508904064:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1789760705.932436:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760705.932475:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508904064:12345-0@lo:13:osp-pre-1-0.0 Request processed in 70us (206us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.932482:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 352 00000100:00100000:2.0:1789760705.932523:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4defdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508904064:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760705.932768:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df82df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508904192:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.932801:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760705.932822:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8912300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508904704:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.932842:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760705.932857:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8910700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508905216:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.932876:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760705.932892:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1789760705.932897:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df82e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508904320:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.932915:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760705.932928:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8910380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508904832:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.932944:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760705.932956:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8911f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508905344:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.932972:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760705.932996:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508904320 00000100:00100000:1.0:1789760705.933000:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508904320 00000100:00100000:1.0:1789760705.933024:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508904832 00000100:00100000:1.0:1789760705.933027:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508904832 00000100:00100000:1.0:1789760705.933041:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508905344 00000100:00100000:1.0:1789760705.933043:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508905344 00000100:00100000:1.0:1789760705.933059:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 845 00000100:00100000:1.0:1789760705.933082:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508904320:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.933115:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508904320:12345-0@lo:400:kworker.0 Request processed in 50us (199us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760705.933122:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 845 00000100:00100000:1.0:1789760705.933134:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 846 00000100:00100000:1.0:1789760705.933139:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f1180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508904832:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.933163:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f1180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508904832:12345-0@lo:400:kworker.0 Request processed in 25us (218us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760705.933169:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 846 00000100:00100000:1.0:1789760705.933178:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 847 00000100:00100000:1.0:1789760705.933183:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508905344:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.933202:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508905344:12345-0@lo:400:kworker.0 Request processed in 19us (229us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760705.933207:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 847 00000100:00100000:1.0:1789760705.933238:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df82e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508904320:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.933247:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8910380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508904832:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.933255:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8911f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508905344:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.933465:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760705.933474:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4defdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508905728:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760705.933499:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760705.933524:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508905728 00000100:00100000:2.0:1789760705.933527:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508905728 00000100:00100000:2.0:1789760705.933554:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 353 00000100:00100000:2.0:1789760705.933560:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508905728:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:2.0:1789760705.933571:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760705.933612:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508905728:12345-0@lo:13:osp-pre-1-1.0 Request processed in 51us (112us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.933618:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 353 00000100:00100000:2.0:1789760705.933633:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df82d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508904448:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.933656:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760705.933683:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8910000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508904960:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.933701:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760705.933714:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8913100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508905472:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.933730:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760705.933743:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [3->2] 00000100:00100000:2.0:1789760705.933748:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8913480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0001_UUID:3480:1876699508904576:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.933764:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760705.933776:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8913b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508905088:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.933793:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760705.933807:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8912680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508905600:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.933822:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760705.933842:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508904704 00000100:00100000:2.0:1789760705.933845:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508904704 00000100:00100000:2.0:1789760705.933866:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508904448 00000100:00100000:2.0:1789760705.933868:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508904448 00000100:00100000:2.0:1789760705.933879:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 411 00000100:00100000:2.0:1789760705.933884:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8911500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508904704:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.933914:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8911500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508904704:12345-0@lo:400:kworker.0 Request processed in 30us (1071us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.933920:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 411 00000100:00100000:2.0:1789760705.933929:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 412 00000100:00100000:2.0:1789760705.933934:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508904448:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.933958:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508904448:12345-0@lo:400:kworker.0 Request processed in 24us (301us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.933963:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 412 00000100:00100000:2.0:1789760705.933977:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508905216 00000100:00100000:2.0:1789760705.933980:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508905216 00000100:00100000:2.0:1789760705.934000:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508905088 00000100:00100000:2.0:1789760705.934002:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508905088 00000100:00100000:2.0:1789760705.934017:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508905600 00000100:00100000:2.0:1789760705.934019:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508905600 00000100:00100000:2.0:1789760705.934029:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 299 00000100:00100000:2.0:1789760705.934034:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508905216:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934059:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508905216:12345-0@lo:400:kworker.0 Request processed in 25us (1182us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.934088:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 299 00000100:00100000:2.0:1789760705.934098:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 300 00000100:00100000:2.0:1789760705.934103:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defde300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508905088:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934125:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defde300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508905088:12345-0@lo:400:kworker.0 Request processed in 23us (332us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.934134:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 300 00000100:00100000:2.0:1789760705.934142:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 301 00000100:00100000:2.0:1789760705.934147:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defded80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508905600:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934166:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defded80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508905600:12345-0@lo:400:kworker.0 Request processed in 19us (343us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.934172:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 301 00000100:00100000:2.0:1789760705.934193:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508904960 00000100:00100000:2.0:1789760705.934196:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508904960 00000100:00100000:2.0:1789760705.934216:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 848 00000100:00100000:2.0:1789760705.934221:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdfb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508904960:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934246:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdfb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508904960:12345-0@lo:400:kworker.0 Request processed in 25us (544us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.934252:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 848 00000100:00100000:2.0:1789760705.934259:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508905472 00000100:00100000:2.0:1789760705.934262:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508905472 00000100:00100000:2.0:1789760705.934277:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508904576 00000100:00100000:2.0:1789760705.934280:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508904576 00000100:00100000:2.0:1789760705.934289:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 849 00000100:00100000:2.0:1789760705.934294:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508905472:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934316:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508905472:12345-0@lo:400:kworker.0 Request processed in 22us (586us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.934321:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 849 00000100:00100000:2.0:1789760705.934330:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 850 00000100:00100000:2.0:1789760705.934335:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defde680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3480:x1876699508904576:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934357:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defde680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3480:x1876699508904576:12345-0@lo:400:kworker.0 Request processed in 21us (592us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760705.934488:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 850 00000100:00100000:2.0:1789760705.934529:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4defdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508905728:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760705.934559:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8913480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0001_UUID:3480:1876699508904576:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934568:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8913b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508905088:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934575:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8912680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508905600:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934595:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df82d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508904448:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934604:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8910000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508904960:0@lo:400:kworker.0 00000100:00100000:2.0:1789760705.934612:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8913100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508905472:0@lo:400:kworker.0 00000100:00080000:0.0:1789760705.935228:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:0.0:1789760705.935233:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:0.0:1789760705.935235:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:0.0:1789760705.935237:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:0.0:1789760705.935250:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508904192 00000100:00100000:0.0:1789760705.935255:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508904192 00000100:00100000:0.0:1789760705.935285:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 464 00000100:00100000:0.0:1789760705.935291:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4d2913850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508904192:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760705.935333:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4d2913850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508904192:12345-0@lo:400:kworker.0 Request processed in 42us (2531us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760705.935340:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 464 00000100:00100000:1.0:1789760705.935939:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df82df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508904192:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.935955:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8912300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508904704:0@lo:400:kworker.0 00000100:00100000:1.0:1789760705.935965:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8910700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508905216:0@lo:400:kworker.0 00000001:00080000:3.0:1789760705.939304:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971870 is committed 00002000:00010000:0.0:1789760705.939510:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18644992 -> 18649088 00010000:00010000:0.0:1789760705.939524:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18644992->18649087] (req 18644992->18649087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.939537:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18644992->18649087] (req 18644992->18649087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.939553:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18644992->18649087] (req 18644992->18649087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.939561:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760705.939566:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18644992->18649087] (req 18644992->18649087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.939576:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18644992->18649087] (req 18644992->18649087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.939606:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18644992->18649087] (req 18644992->18649087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.939614:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207635 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18644992->18649087] (req 18644992->18649087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760705.939706:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484142976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31757us (32079us total) trans 4294971870 rc 0/0 00000100:00100000:0.0:1789760705.939716:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4581 00000100:00100000:0.0:1789760705.939724:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695207347784 time 695238863526 state 0x2 ref 0 00000100:00100000:3.0:1789760705.943233:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760705.943643:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484143104 00000100:00100000:0.0:1789760705.943652:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484143104 00000100:00100000:0.0:1789760705.943767:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695207347784 time 695238863526 state 0x0 ref 1 00000100:00100000:0.0:1789760705.943781:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695239123942 time 695238863526 state 0x1 ref 1 00000100:00100000:0.0:1789760705.943839:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695239123942 time 695274769584 state 0x0 ref 1 00000100:00100000:0.0:1789760705.943852:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484143104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760705.943897:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18649088, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:0.0:1789760705.943905:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.943920:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:0.0:1789760705.943925:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760705.943930:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18649088->18653183] (req 18649088->18653183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760705.943944:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18649088->18653183] (req 18649088->18653183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760705.967575:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971871 is committed 00002000:00010000:1.0:1789760705.967665:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18649088 -> 18653184 00010000:00010000:1.0:1789760705.967738:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18649088->18653183] (req 18649088->18653183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.967751:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18649088->18653183] (req 18649088->18653183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.967760:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18649088->18653183] (req 18649088->18653183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.967773:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760705.967778:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18649088->18653183] (req 18649088->18653183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.967788:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18649088->18653183] (req 18649088->18653183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.967807:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18649088->18653183] (req 18649088->18653183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.967814:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620763c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18649088->18653183] (req 18649088->18653183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760705.967902:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484143104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24047us (24672us total) trans 4294971871 rc 0/0 00000100:00100000:1.0:1789760705.967913:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4582 00000100:00100000:1.0:1789760705.967922:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695239123942 time 695274769584 state 0x2 ref 0 00000100:00100000:2.0:1789760705.971266:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760705.971329:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484143232 00000100:00100000:1.0:1789760705.971338:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484143232 00000100:00100000:1.0:1789760705.971403:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695239123942 time 695274769584 state 0x0 ref 1 00000100:00100000:1.0:1789760705.971418:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695275030000 time 695274769584 state 0x1 ref 1 00000100:00100000:1.0:1789760705.971449:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695275030000 time 695302380744 state 0x0 ref 1 00000100:00100000:1.0:1789760705.971465:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484143232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760705.971509:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18653184, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:1.0:1789760705.971518:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.971533:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:1.0:1789760705.971537:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760705.971543:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18653184->18657279] (req 18653184->18657279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760705.971556:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18653184->18657279] (req 18653184->18657279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760706.013988:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18653184 -> 18657280 00010000:00010000:2.0:1789760706.014008:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18653184->18657279] (req 18653184->18657279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.014022:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18653184->18657279] (req 18653184->18657279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.014032:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18653184->18657279] (req 18653184->18657279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.014040:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760706.014044:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18653184->18657279] (req 18653184->18657279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.014059:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18653184->18657279] (req 18653184->18657279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.014098:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18653184->18657279] (req 18653184->18657279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.014106:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207643 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18653184->18657279] (req 18653184->18657279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760706.014206:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484143232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42738us (42944us total) trans 4294971872 rc 0/0 00000100:00100000:2.0:1789760706.014217:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4583 00000100:00100000:2.0:1789760706.014227:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695275030000 time 695302380744 state 0x2 ref 0 00000001:00080000:3.0:1789760706.014329:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971872 is committed 00000100:00100000:2.0:1789760706.018686:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760706.018816:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484143360 00000100:00100000:3.0:1789760706.018827:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484143360 00000100:00100000:3.0:1789760706.018897:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695275030000 time 695302380744 state 0x0 ref 1 00000100:00100000:3.0:1789760706.018913:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695302641160 time 695302380744 state 0x1 ref 1 00000100:00100000:3.0:1789760706.018943:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695302641160 time 695349874284 state 0x0 ref 1 00000100:00100000:3.0:1789760706.018957:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484143360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760706.019001:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18657280, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:3.0:1789760706.019010:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.019025:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:3.0:1789760706.019030:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760706.019037:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18657280->18661375] (req 18657280->18661375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.019052:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18657280->18661375] (req 18657280->18661375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1789760706.019529:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:1.0:1789760706.052321:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971873 is committed 00002000:00010000:3.0:1789760706.052479:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18657280 -> 18661376 00010000:00010000:3.0:1789760706.052497:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18657280->18661375] (req 18657280->18661375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.052515:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18657280->18661375] (req 18657280->18661375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.052524:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18657280->18661375] (req 18657280->18661375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.052531:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760706.052536:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18657280->18661375] (req 18657280->18661375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.052545:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18657280->18661375] (req 18657280->18661375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.052563:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18657280->18661375] (req 18657280->18661375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.052570:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620764a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18657280->18661375] (req 18657280->18661375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760706.052924:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484143360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33964us (34253us total) trans 4294971873 rc 0/0 00000100:00100000:3.0:1789760706.052934:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4584 00000100:00100000:3.0:1789760706.052944:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695302641160 time 695349874284 state 0x2 ref 0 00000100:00100000:1.0:1789760706.058778:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760706.059296:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484143488 00000100:00100000:2.0:1789760706.059306:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484143488 00000100:00100000:2.0:1789760706.059381:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695302641160 time 695349874284 state 0x0 ref 1 00000100:00100000:2.0:1789760706.059407:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695350134700 time 695349874284 state 0x1 ref 1 00000100:00100000:2.0:1789760706.059874:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695350134700 time 695390802574 state 0x0 ref 1 00000100:00100000:2.0:1789760706.059890:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7073050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484143488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760706.059937:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18661376, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:2.0:1789760706.059964:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.059981:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:2.0:1789760706.059986:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760706.059993:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18661376->18665471] (req 18661376->18665471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.060006:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18661376->18665471] (req 18661376->18665471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760706.096374:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18661376 -> 18665472 00010000:00010000:3.0:1789760706.096394:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18661376->18665471] (req 18661376->18665471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.096407:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18661376->18665471] (req 18661376->18665471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.096416:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18661376->18665471] (req 18661376->18665471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.096423:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760706.096428:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18661376->18665471] (req 18661376->18665471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.096436:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18661376->18665471] (req 18661376->18665471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.096454:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18661376->18665471] (req 18661376->18665471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.096461:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207651 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18661376->18665471] (req 18661376->18665471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760706.096552:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7073050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484143488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36662us (37893us total) trans 4294971874 rc 0/0 00000100:00100000:3.0:1789760706.096562:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4585 00000100:00100000:3.0:1789760706.096572:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695350134700 time 695390802574 state 0x2 ref 0 00000001:00080000:1.0:1789760706.097484:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971874 is committed 00000100:00100000:2.0:1789760706.101363:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760706.101502:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484143616 00000100:00100000:1.0:1789760706.101513:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484143616 00000100:00100000:1.0:1789760706.101567:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695350134700 time 695390802574 state 0x0 ref 1 00000100:00100000:1.0:1789760706.101594:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695391062990 time 695390802574 state 0x1 ref 1 00000100:00100000:1.0:1789760706.101657:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695391062990 time 695432586676 state 0x0 ref 1 00000100:00100000:1.0:1789760706.101685:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484143616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760706.101732:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18665472, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:1.0:1789760706.101740:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.101756:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:1.0:1789760706.101761:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760706.101768:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18665472->18669567] (req 18665472->18669567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.101783:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18665472->18669567] (req 18665472->18669567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760706.145108:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18665472 -> 18669568 00010000:00010000:1.0:1789760706.145127:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18665472->18669567] (req 18665472->18669567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.145141:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18665472->18669567] (req 18665472->18669567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.145150:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18665472->18669567] (req 18665472->18669567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.145158:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760706.145163:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18665472->18669567] (req 18665472->18669567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.145171:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18665472->18669567] (req 18665472->18669567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.145189:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18665472->18669567] (req 18665472->18669567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.145197:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207658 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18665472->18669567] (req 18665472->18669567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760706.145288:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484143616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43603us (43931us total) trans 4294971875 rc 0/0 00000100:00100000:1.0:1789760706.145298:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4586 00000100:00100000:1.0:1789760706.145306:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695391062990 time 695432586676 state 0x2 ref 0 00000001:00080000:3.0:1789760706.145807:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971875 is committed 00000100:00100000:2.0:1789760706.149256:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760706.149334:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484143744 00000100:00100000:1.0:1789760706.149343:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484143744 00000100:00100000:1.0:1789760706.149403:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695391062990 time 695432586676 state 0x0 ref 1 00000100:00100000:1.0:1789760706.149417:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695432847092 time 695432586676 state 0x1 ref 1 00000100:00100000:1.0:1789760706.149442:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695432847092 time 695480373202 state 0x0 ref 1 00000100:00100000:1.0:1789760706.149453:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484143744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760706.149502:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18669568, mtime: 0x6aad94c1, atime: 0x0, ctime: 0x6aad94c1, blocks: 0x9007 00010000:00010000:1.0:1789760706.149509:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.149525:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:1.0:1789760706.149530:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760706.149537:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18669568->18673663] (req 18669568->18673663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.149551:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18669568->18673663] (req 18669568->18673663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760706.194642:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18669568 -> 18673664 00002000:00010000:1.0:1789760706.194651:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760705 -> 1789760706 00002000:00010000:1.0:1789760706.194654:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760705 -> 1789760706 00010000:00010000:1.0:1789760706.194675:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18669568->18673663] (req 18669568->18673663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.194689:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18669568->18673663] (req 18669568->18673663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.194698:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18669568->18673663] (req 18669568->18673663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.194706:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760706.194711:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18669568->18673663] (req 18669568->18673663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.194720:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18669568->18673663] (req 18669568->18673663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.194739:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18669568->18673663] (req 18669568->18673663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.194748:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620765f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18669568->18673663] (req 18669568->18673663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760706.194839:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484143744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45382us (45585us total) trans 4294971876 rc 0/0 00000100:00100000:1.0:1789760706.194850:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4587 00000100:00100000:1.0:1789760706.194860:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695432847092 time 695480373202 state 0x2 ref 0 00000100:00100000:1.0:1789760706.199191:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760706.199319:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484143872 00000100:00100000:3.0:1789760706.199329:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484143872 00000100:00100000:3.0:1789760706.199381:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695432847092 time 695480373202 state 0x0 ref 1 00000100:00100000:3.0:1789760706.199396:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695480633618 time 695480373202 state 0x1 ref 1 00000100:00100000:3.0:1789760706.199426:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695480633618 time 695530356952 state 0x0 ref 1 00000100:00100000:3.0:1789760706.199440:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484143872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760706.199486:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18673664, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:3.0:1789760706.199494:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.199573:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:3.0:1789760706.199593:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760706.199601:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18673664->18677759] (req 18673664->18677759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.199615:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18673664->18677759] (req 18673664->18677759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760706.201801:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971876 is committed 00000001:00080000:2.0:1789760706.243921:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971877 is committed 00002000:00010000:3.0:1789760706.244008:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18673664 -> 18677760 00010000:00010000:3.0:1789760706.244022:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18673664->18677759] (req 18673664->18677759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.244040:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18673664->18677759] (req 18673664->18677759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.244050:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18673664->18677759] (req 18673664->18677759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.244058:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760706.244062:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18673664->18677759] (req 18673664->18677759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.244101:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18673664->18677759] (req 18673664->18677759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.244121:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18673664->18677759] (req 18673664->18677759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.244129:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207666 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18673664->18677759] (req 18673664->18677759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760706.244219:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484143872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44777us (45034us total) trans 4294971877 rc 0/0 00000100:00100000:3.0:1789760706.244230:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4588 00000100:00100000:3.0:1789760706.244240:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695480633618 time 695530356952 state 0x2 ref 0 00000100:00100000:2.0:1789760706.249653:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760706.249789:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484144000 00000100:00100000:3.0:1789760706.249800:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484144000 00000100:00100000:3.0:1789760706.249849:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695480633618 time 695530356952 state 0x0 ref 1 00000100:00100000:3.0:1789760706.249863:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695530617368 time 695530356952 state 0x1 ref 1 00000100:00100000:3.0:1789760706.249895:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695530617368 time 695580825555 state 0x0 ref 1 00000100:00100000:3.0:1789760706.249908:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484144000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760706.249956:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18677760, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:3.0:1789760706.249966:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.249982:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:3.0:1789760706.249986:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760706.249992:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18677760->18681855] (req 18677760->18681855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.250006:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18677760->18681855] (req 18677760->18681855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760706.311215:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971878 is committed 00002000:00010000:0.0:1789760706.311332:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18677760 -> 18681856 00010000:00010000:0.0:1789760706.311349:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18677760->18681855] (req 18677760->18681855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.311431:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18677760->18681855] (req 18677760->18681855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.311441:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18677760->18681855] (req 18677760->18681855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.311449:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760706.311455:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18677760->18681855] (req 18677760->18681855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.311465:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18677760->18681855] (req 18677760->18681855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.311486:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18677760->18681855] (req 18677760->18681855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.311493:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620766d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18677760->18681855] (req 18677760->18681855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760706.311631:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484144000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61720us (61984us total) trans 4294971878 rc 0/0 00000100:00100000:0.0:1789760706.311642:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4589 00000100:00100000:0.0:1789760706.311652:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695530617368 time 695580825555 state 0x2 ref 0 00000100:00100000:2.0:1789760706.327505:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760706.327611:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484144128 00000100:00100000:1.0:1789760706.327621:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484144128 00000100:00100000:1.0:1789760706.327667:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695530617368 time 695580825555 state 0x0 ref 1 00000100:00100000:1.0:1789760706.327681:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695581085971 time 695580825555 state 0x1 ref 1 00000100:00100000:1.0:1789760706.327799:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695581085971 time 695658728551 state 0x0 ref 1 00000100:00100000:1.0:1789760706.327812:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484144128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760706.327856:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18681856, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:1.0:1789760706.327865:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.327879:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:1.0:1789760706.327884:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760706.327890:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18681856->18685951] (req 18681856->18685951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.327904:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18681856->18685951] (req 18681856->18685951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760706.374764:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971879 is committed 00002000:00010000:0.0:1789760706.375006:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18681856 -> 18685952 00010000:00010000:0.0:1789760706.375021:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18681856->18685951] (req 18681856->18685951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.375033:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18681856->18685951] (req 18681856->18685951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.375043:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18681856->18685951] (req 18681856->18685951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.375055:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760706.375059:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18681856->18685951] (req 18681856->18685951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.375106:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18681856->18685951] (req 18681856->18685951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.375126:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18681856->18685951] (req 18681856->18685951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.375134:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207674 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18681856->18685951] (req 18681856->18685951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760706.375209:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484144128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47395us (47708us total) trans 4294971879 rc 0/0 00000100:00100000:0.0:1789760706.375218:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4590 00000100:00100000:0.0:1789760706.375227:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695581085971 time 695658728551 state 0x2 ref 0 00000100:00100000:1.0:1789760706.377935:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760706.378111:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484144256 00000100:00100000:2.0:1789760706.378120:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484144256 00000100:00100000:2.0:1789760706.378176:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695581085971 time 695658728551 state 0x0 ref 1 00000100:00100000:2.0:1789760706.378191:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695658988967 time 695658728551 state 0x1 ref 1 00000100:00100000:2.0:1789760706.378214:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695658988967 time 695709145442 state 0x0 ref 1 00000100:00100000:2.0:1789760706.378225:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484144256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760706.378268:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18685952, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:2.0:1789760706.378276:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.378292:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:2.0:1789760706.378296:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760706.378311:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18685952->18690047] (req 18685952->18690047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.378326:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18685952->18690047] (req 18685952->18690047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760706.423890:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18685952 -> 18690048 00010000:00010000:2.0:1789760706.423909:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18685952->18690047] (req 18685952->18690047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.423927:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18685952->18690047] (req 18685952->18690047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.423937:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18685952->18690047] (req 18685952->18690047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.423947:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760706.423952:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18685952->18690047] (req 18685952->18690047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.423961:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18685952->18690047] (req 18685952->18690047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.423986:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18685952->18690047] (req 18685952->18690047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.423994:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620767b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18685952->18690047] (req 18685952->18690047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760706.424100:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484144256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45872us (46168us total) trans 4294971880 rc 0/0 00000100:00100000:2.0:1789760706.424110:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4591 00000100:00100000:2.0:1789760706.424120:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695658988967 time 695709145442 state 0x2 ref 0 00000001:00080000:0.0:1789760706.424218:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971880 is committed 00000100:00100000:2.0:1789760706.428556:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760706.430294:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484144384 00000100:00100000:3.0:1789760706.430307:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484144384 00000100:00100000:3.0:1789760706.430366:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695658988967 time 695709145442 state 0x0 ref 1 00000100:00100000:3.0:1789760706.430382:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695709405858 time 695709145442 state 0x1 ref 1 00000100:00100000:3.0:1789760706.430403:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695709405858 time 695761333709 state 0x0 ref 1 00000100:00100000:3.0:1789760706.430416:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484144384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760706.430464:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18690048, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:3.0:1789760706.430473:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.430488:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:3.0:1789760706.430493:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760706.430499:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18690048->18694143] (req 18690048->18694143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.430513:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18690048->18694143] (req 18690048->18694143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.446080:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760706.466910:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971881 is committed 00002000:00010000:1.0:1789760706.466993:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18690048 -> 18694144 00010000:00010000:1.0:1789760706.467009:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18690048->18694143] (req 18690048->18694143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.467021:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18690048->18694143] (req 18690048->18694143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.467031:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18690048->18694143] (req 18690048->18694143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.467040:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760706.467046:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18690048->18694143] (req 18690048->18694143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.467056:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18690048->18694143] (req 18690048->18694143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.467098:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18690048->18694143] (req 18690048->18694143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.467107:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207682 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18690048->18694143] (req 18690048->18694143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760706.467196:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484144384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36777us (38643us total) trans 4294971881 rc 0/0 00000100:00100000:1.0:1789760706.467205:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4592 00000100:00100000:1.0:1789760706.467215:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695709405858 time 695761333709 state 0x2 ref 0 00000100:00100000:1.0:1789760706.482214:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760706.482366:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484144512 00000100:00100000:0.0:1789760706.482376:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484144512 00000100:00100000:0.0:1789760706.482431:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695709405858 time 695761333709 state 0x0 ref 1 00000100:00100000:0.0:1789760706.482447:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695761594125 time 695761333709 state 0x1 ref 1 00000100:00100000:0.0:1789760706.482484:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695761594125 time 695813415916 state 0x0 ref 1 00000100:00100000:0.0:1789760706.482497:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484144512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760706.482594:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18694144, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:0.0:1789760706.482604:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.482620:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:0.0:1789760706.482624:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760706.482631:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18694144->18698239] (req 18694144->18698239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.482645:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18694144->18698239] (req 18694144->18698239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760706.519226:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18694144 -> 18698240 00010000:00010000:3.0:1789760706.519245:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18694144->18698239] (req 18694144->18698239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.519258:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18694144->18698239] (req 18694144->18698239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.519268:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18694144->18698239] (req 18694144->18698239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.519276:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760706.519281:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18694144->18698239] (req 18694144->18698239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.519290:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18694144->18698239] (req 18694144->18698239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.519310:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18694144->18698239] (req 18694144->18698239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.519318:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207689 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18694144->18698239] (req 18694144->18698239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760706.519429:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484144512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36929us (37219us total) trans 4294971882 rc 0/0 00000100:00100000:3.0:1789760706.519439:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4593 00000100:00100000:3.0:1789760706.519449:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695761594125 time 695813415916 state 0x2 ref 0 00000001:00080000:2.0:1789760706.521766:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971882 is committed 00000100:00100000:2.0:1789760706.536537:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760706.536764:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484144640 00000100:00100000:0.0:1789760706.536775:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484144640 00000100:00100000:0.0:1789760706.536836:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695761594125 time 695813415916 state 0x0 ref 1 00000100:00100000:0.0:1789760706.536922:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695813676332 time 695813415916 state 0x1 ref 1 00000100:00100000:0.0:1789760706.537016:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695813676332 time 695867945618 state 0x0 ref 1 00000100:00100000:0.0:1789760706.537032:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484144640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760706.537114:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18698240, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:0.0:1789760706.537123:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.537153:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:0.0:1789760706.537157:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760706.537164:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18698240->18702335] (req 18698240->18702335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.537179:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18698240->18702335] (req 18698240->18702335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760706.597765:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971883 is committed 00002000:00010000:2.0:1789760706.597985:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18698240 -> 18702336 00010000:00010000:2.0:1789760706.598002:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18698240->18702335] (req 18698240->18702335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.598015:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18698240->18702335] (req 18698240->18702335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.598024:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18698240->18702335] (req 18698240->18702335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.598032:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760706.598038:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18698240->18702335] (req 18698240->18702335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.598047:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18698240->18702335] (req 18698240->18702335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.598095:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18698240->18702335] (req 18698240->18702335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.598104:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207690 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18698240->18702335] (req 18698240->18702335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760706.598193:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484144640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61160us (61661us total) trans 4294971883 rc 0/0 00000100:00100000:2.0:1789760706.598204:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4594 00000100:00100000:2.0:1789760706.598214:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695813676332 time 695867945618 state 0x2 ref 0 00000100:00100000:1.0:1789760706.606662:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760706.606899:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484144768 00000100:00100000:2.0:1789760706.606910:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484144768 00000100:00100000:2.0:1789760706.606965:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695813676332 time 695867945618 state 0x0 ref 1 00000100:00100000:2.0:1789760706.606982:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695868206034 time 695867945618 state 0x1 ref 1 00000100:00100000:2.0:1789760706.607012:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695868206034 time 695937943370 state 0x0 ref 1 00000100:00100000:2.0:1789760706.607025:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484144768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760706.607110:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18702336, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:2.0:1789760706.607119:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.607144:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=695s v=(0 0) 00010000:00010000:2.0:1789760706.607148:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760706.607155:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18702336->18706431] (req 18702336->18706431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.607169:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18702336->18706431] (req 18702336->18706431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760706.697773:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971884 is committed 00002000:00010000:2.0:1789760706.700913:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18702336 -> 18706432 00010000:00010000:2.0:1789760706.700935:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18702336->18706431] (req 18702336->18706431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.700949:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18702336->18706431] (req 18702336->18706431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.700959:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18702336->18706431] (req 18702336->18706431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.700966:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760706.700971:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18702336->18706431] (req 18702336->18706431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.700980:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18702336->18706431] (req 18702336->18706431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.701008:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18702336->18706431] (req 18702336->18706431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760706.701016:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207697 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18702336->18706431] (req 18702336->18706431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760706.701617:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484144768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 94580us (94950us total) trans 4294971884 rc 0/0 00000100:00100000:2.0:1789760706.701627:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4595 00000100:00100000:2.0:1789760706.701638:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695868206034 time 695937943370 state 0x2 ref 0 00000100:00100000:2.0:1789760706.715028:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760706.718022:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484144896 00000100:00100000:1.0:1789760706.718039:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484144896 00000100:00100000:0.0:1789760706.718256:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695868206034 time 695937943370 state 0x0 ref 1 00000100:00100000:0.0:1789760706.718272:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695938203786 time 695937943370 state 0x1 ref 1 00000100:00100000:0.0:1789760706.718300:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695938203786 time 696049231341 state 0x0 ref 1 00000100:00100000:0.0:1789760706.718315:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484144896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760706.718386:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18706432, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:0.0:1789760706.718396:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.718411:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:0.0:1789760706.718416:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760706.718424:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18706432->18710527] (req 18706432->18710527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760706.718439:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18706432->18710527] (req 18706432->18710527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760706.787120:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18706432 -> 18710528 00010000:00010000:3.0:1789760706.787140:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18706432->18710527] (req 18706432->18710527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.787153:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18706432->18710527] (req 18706432->18710527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.787162:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18706432->18710527] (req 18706432->18710527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.787168:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760706.787173:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18706432->18710527] (req 18706432->18710527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.787182:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18706432->18710527] (req 18706432->18710527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.787201:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18706432->18710527] (req 18706432->18710527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.787209:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620769e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18706432->18710527] (req 18706432->18710527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760706.787315:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484144896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69000us (72294us total) trans 4294971885 rc 0/0 00000100:00100000:3.0:1789760706.787326:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4596 00000100:00100000:3.0:1789760706.787335:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695938203786 time 696049231341 state 0x2 ref 0 00000001:00080000:1.0:1789760706.789254:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971885 is committed 00000100:00100000:1.0:1789760706.791811:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760706.791862:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760706.791901:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760706.791928:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760706.791946:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760706.792112:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484145152 00000100:00100000:3.0:1789760706.792122:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484145152 00000100:00100000:3.0:1789760706.792182:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484145408 00000100:00100000:3.0:1789760706.792184:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484145408 00000100:00100000:3.0:1789760706.792202:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484145280 00000100:00100000:3.0:1789760706.792205:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484145280 00000100:00100000:3.0:1789760706.792218:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484145536 00000100:00100000:3.0:1789760706.792220:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484145536 00000100:00100000:3.0:1789760706.792236:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 851 00000100:00100000:3.0:1789760706.792246:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484145152:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760706.792323:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484145152:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 78us (521us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760706.792332:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 851 00000100:00100000:3.0:1789760706.792349:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 852 00000100:00100000:3.0:1789760706.792355:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484145408:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760706.792392:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484145408:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 37us (529us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760706.792399:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 852 00000100:00100000:3.0:1789760706.792411:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 853 00000100:00100000:3.0:1789760706.792417:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484145280:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760706.792439:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484145280:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 23us (511us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760706.792444:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 853 00000100:00100000:3.0:1789760706.792452:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 854 00000100:00100000:3.0:1789760706.792457:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f1500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484145536:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760706.792475:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f1500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484145536:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 18us (529us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760706.792481:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 854 00000100:00080000:3.0:1789760706.792494:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:3.0:1789760706.792498:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:0.0:1789760706.792715:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484145024 00000100:00100000:0.0:1789760706.792719:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484145024 00000100:00100000:0.0:1789760706.792762:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 465 00000100:00100000:0.0:1789760706.792771:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4d2913050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484145024:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760706.792802:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4d2913050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484145024:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 34us (901us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760706.792808:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 465 00000100:00100000:1.0:1789760706.797123:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760706.797404:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484145664 00000100:00100000:3.0:1789760706.797414:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484145664 00000100:00100000:3.0:1789760706.797467:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 695938203786 time 696049231341 state 0x0 ref 1 00000100:00100000:3.0:1789760706.797482:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696049491757 time 696049231341 state 0x1 ref 1 00000100:00100000:3.0:1789760706.797516:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696049491757 time 696128447314 state 0x0 ref 1 00000100:00100000:3.0:1789760706.797528:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484145664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760706.797639:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18710528, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:3.0:1789760706.797648:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.797673:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:3.0:1789760706.797678:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760706.797684:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18710528->18714623] (req 18710528->18714623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.797698:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18710528->18714623] (req 18710528->18714623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760706.884784:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971886 is committed 00002000:00010000:1.0:1789760706.884903:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18710528 -> 18714624 00010000:00010000:1.0:1789760706.884919:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18710528->18714623] (req 18710528->18714623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.884933:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18710528->18714623] (req 18710528->18714623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.884943:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18710528->18714623] (req 18710528->18714623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.884954:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760706.884959:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18710528->18714623] (req 18710528->18714623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.884968:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18710528->18714623] (req 18710528->18714623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.884987:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18710528->18714623] (req 18710528->18714623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.884995:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962076a5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18710528->18714623] (req 18710528->18714623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760706.885120:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484145664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 87590us (88002us total) trans 4294971886 rc 0/0 00000100:00100000:1.0:1789760706.885131:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4597 00000100:00100000:1.0:1789760706.885141:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696049491757 time 696128447314 state 0x2 ref 0 00000100:00100000:2.0:1789760706.888913:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760706.889234:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484145792 00000100:00100000:3.0:1789760706.889245:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484145792 00000100:00100000:3.0:1789760706.889299:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696049491757 time 696128447314 state 0x0 ref 1 00000100:00100000:3.0:1789760706.889313:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696128707730 time 696128447314 state 0x1 ref 1 00000100:00100000:3.0:1789760706.889344:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696128707730 time 696220275152 state 0x0 ref 1 00000100:00100000:3.0:1789760706.889366:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484145792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760706.889415:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18714624, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:3.0:1789760706.889426:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.889442:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:3.0:1789760706.889446:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760706.889453:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18714624->18718719] (req 18714624->18718719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.889467:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18714624->18718719] (req 18714624->18718719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760706.923591:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971887 is committed 00002000:00010000:3.0:1789760706.923790:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18714624 -> 18718720 00010000:00010000:3.0:1789760706.923805:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18714624->18718719] (req 18714624->18718719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.923818:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18714624->18718719] (req 18714624->18718719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.923827:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18714624->18718719] (req 18714624->18718719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.923833:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760706.923838:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18714624->18718719] (req 18714624->18718719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.923847:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18714624->18718719] (req 18714624->18718719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.923872:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18714624->18718719] (req 18714624->18718719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760706.923879:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076ac lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18714624->18718719] (req 18714624->18718719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760706.924005:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484145792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34646us (35097us total) trans 4294971887 rc 0/0 00000100:00100000:3.0:1789760706.924015:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4598 00000100:00100000:3.0:1789760706.924026:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696128707730 time 696220275152 state 0x2 ref 0 00000100:00100000:2.0:1789760706.932299:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760706.932525:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484145920 00000100:00100000:1.0:1789760706.932536:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484145920 00000100:00100000:1.0:1789760706.932634:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696128707730 time 696220275152 state 0x0 ref 1 00000100:00100000:1.0:1789760706.932650:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696220535568 time 696220275152 state 0x1 ref 1 00000100:00100000:1.0:1789760706.932687:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696220535568 time 696263617825 state 0x0 ref 1 00000100:00100000:1.0:1789760706.932713:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484145920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760706.932761:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18718720, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:1.0:1789760706.932770:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.932783:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:1.0:1789760706.932786:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760706.932791:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18718720->18722815] (req 18718720->18722815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760706.932801:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18718720->18722815] (req 18718720->18722815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760707.000537:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18718720 -> 18722816 00010000:00010000:0.0:1789760707.000556:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18718720->18722815] (req 18718720->18722815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.000569:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18718720->18722815] (req 18718720->18722815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.000590:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18718720->18722815] (req 18718720->18722815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.000597:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760707.000602:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18718720->18722815] (req 18718720->18722815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.000616:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18718720->18722815] (req 18718720->18722815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.000635:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18718720->18722815] (req 18718720->18722815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.000644:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962076b3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18718720->18722815] (req 18718720->18722815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760707.000743:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484145920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68029us (68449us total) trans 4294971888 rc 0/0 00000100:00100000:0.0:1789760707.000754:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4599 00000100:00100000:0.0:1789760707.000763:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696220535568 time 696263617825 state 0x2 ref 0 00000001:00080000:0.0:1789760707.000801:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971888 is committed 00000100:00100000:2.0:1789760707.006711:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760707.006826:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484146048 00000100:00100000:0.0:1789760707.006836:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484146048 00000100:00100000:0.0:1789760707.006890:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696220535568 time 696263617825 state 0x0 ref 1 00000100:00100000:0.0:1789760707.006905:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696263878241 time 696263617825 state 0x1 ref 1 00000100:00100000:0.0:1789760707.006936:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696263878241 time 696337867453 state 0x0 ref 1 00000100:00100000:0.0:1789760707.006950:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484146048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760707.007009:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18722816, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:0.0:1789760707.007018:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.007033:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:0.0:1789760707.007038:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760707.007045:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18722816->18726911] (req 18722816->18726911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.007059:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18722816->18726911] (req 18722816->18726911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760707.034269:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18722816 -> 18726912 00010000:00010000:1.0:1789760707.034289:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18722816->18726911] (req 18722816->18726911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.034302:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18722816->18726911] (req 18722816->18726911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.034309:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18722816->18726911] (req 18722816->18726911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.034317:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760707.034321:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18722816->18726911] (req 18722816->18726911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.034330:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18722816->18726911] (req 18722816->18726911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.034350:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18722816->18726911] (req 18722816->18726911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.034373:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962076ba lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18722816->18726911] (req 18722816->18726911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760707.034471:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484146048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27519us (27764us total) trans 4294971889 rc 0/0 00000100:00100000:1.0:1789760707.034482:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4600 00000100:00100000:1.0:1789760707.034491:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696263878241 time 696337867453 state 0x2 ref 0 00000001:00080000:0.0:1789760707.034697:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971889 is committed 00000100:00100000:1.0:1789760707.039732:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760707.039919:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484146176 00000100:00100000:0.0:1789760707.039930:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484146176 00000100:00100000:0.0:1789760707.039991:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696263878241 time 696337867453 state 0x0 ref 1 00000100:00100000:0.0:1789760707.040007:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696338127869 time 696337867453 state 0x1 ref 1 00000100:00100000:0.0:1789760707.040038:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696338127869 time 696370969302 state 0x0 ref 1 00000100:00100000:0.0:1789760707.040054:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484146176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760707.040137:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18726912, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:0.0:1789760707.040146:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.040161:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:0.0:1789760707.040166:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760707.040172:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18726912->18731007] (req 18726912->18731007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.040187:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18726912->18731007] (req 18726912->18731007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760707.081005:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971890 is committed 00002000:00010000:0.0:1789760707.081128:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18726912 -> 18731008 00010000:00010000:0.0:1789760707.081143:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18726912->18731007] (req 18726912->18731007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.081156:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18726912->18731007] (req 18726912->18731007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.081165:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18726912->18731007] (req 18726912->18731007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.081172:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760707.081177:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18726912->18731007] (req 18726912->18731007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.081186:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18726912->18731007] (req 18726912->18731007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.081205:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18726912->18731007] (req 18726912->18731007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.081212:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962076c1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18726912->18731007] (req 18726912->18731007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760707.081363:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484146176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41303us (41630us total) trans 4294971890 rc 0/0 00000100:00100000:0.0:1789760707.081479:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4601 00000100:00100000:0.0:1789760707.081490:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696338127869 time 696370969302 state 0x2 ref 0 00000100:00100000:2.0:1789760707.084882:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760707.084957:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484146304 00000100:00100000:3.0:1789760707.084967:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484146304 00000100:00100000:3.0:1789760707.085023:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696338127869 time 696370969302 state 0x0 ref 1 00000100:00100000:3.0:1789760707.085038:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696371229718 time 696370969302 state 0x1 ref 1 00000100:00100000:3.0:1789760707.085170:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696371229718 time 696416099557 state 0x0 ref 1 00000100:00100000:3.0:1789760707.085183:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484146304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760707.085230:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18731008, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:3.0:1789760707.085238:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.085260:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:3.0:1789760707.085264:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760707.085270:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18731008->18735103] (req 18731008->18735103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.085283:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18731008->18735103] (req 18731008->18735103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760707.114251:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18731008 -> 18735104 00010000:00010000:2.0:1789760707.114273:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18731008->18735103] (req 18731008->18735103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.114287:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18731008->18735103] (req 18731008->18735103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.114296:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18731008->18735103] (req 18731008->18735103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.114304:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760707.114310:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18731008->18735103] (req 18731008->18735103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.114318:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18731008->18735103] (req 18731008->18735103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.114336:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18731008->18735103] (req 18731008->18735103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.114343:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962076c8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18731008->18735103] (req 18731008->18735103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760707.114459:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484146304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29273us (29582us total) trans 4294971891 rc 0/0 00000100:00100000:2.0:1789760707.114469:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4602 00000100:00100000:2.0:1789760707.114479:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696371229718 time 696416099557 state 0x2 ref 0 00000001:00080000:1.0:1789760707.117229:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971891 is committed 00000100:00100000:2.0:1789760707.119154:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760707.119247:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484146432 00000100:00100000:0.0:1789760707.119258:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484146432 00000100:00100000:0.0:1789760707.119322:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696371229718 time 696416099557 state 0x0 ref 1 00000100:00100000:0.0:1789760707.119336:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696416359973 time 696416099557 state 0x1 ref 1 00000100:00100000:0.0:1789760707.119512:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696416359973 time 696450441293 state 0x0 ref 1 00000100:00100000:0.0:1789760707.119527:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484146432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760707.119584:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18735104, mtime: 0x6aad94c2, atime: 0x0, ctime: 0x6aad94c2, blocks: 0x9007 00010000:00010000:0.0:1789760707.119593:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.119608:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:0.0:1789760707.119612:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760707.119618:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18735104->18739199] (req 18735104->18739199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.119632:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18735104->18739199] (req 18735104->18739199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760707.158897:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18735104 -> 18739200 00002000:00010000:0.0:1789760707.158905:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760706 -> 1789760707 00002000:00010000:0.0:1789760707.158908:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760706 -> 1789760707 00010000:00010000:0.0:1789760707.158921:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18735104->18739199] (req 18735104->18739199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.158934:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18735104->18739199] (req 18735104->18739199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.158944:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18735104->18739199] (req 18735104->18739199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.158952:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760707.158957:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18735104->18739199] (req 18735104->18739199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.158965:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18735104->18739199] (req 18735104->18739199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.158984:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18735104->18739199] (req 18735104->18739199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.158991:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962076cf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18735104->18739199] (req 18735104->18739199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760707.159105:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484146432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39577us (39955us total) trans 4294971892 rc 0/0 00000100:00100000:0.0:1789760707.159114:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4603 00000100:00100000:0.0:1789760707.159123:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696416359973 time 696450441293 state 0x2 ref 0 00000001:00080000:1.0:1789760707.162217:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971892 is committed 00000100:00100000:0.0:1789760707.162564:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760707.162649:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484146560 00000100:00100000:1.0:1789760707.162718:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484146560 00000100:00100000:1.0:1789760707.162769:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696416359973 time 696450441293 state 0x0 ref 1 00000100:00100000:1.0:1789760707.162782:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696450701709 time 696450441293 state 0x1 ref 1 00000100:00100000:1.0:1789760707.162848:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696450701709 time 696493778279 state 0x0 ref 1 00000100:00100000:1.0:1789760707.162860:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484146560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760707.162906:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18739200, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:1.0:1789760707.162916:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.162931:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:1.0:1789760707.162935:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760707.162941:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18739200->18743295] (req 18739200->18743295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.162953:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18739200->18743295] (req 18739200->18743295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760707.198029:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18739200 -> 18743296 00010000:00010000:1.0:1789760707.198049:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18739200->18743295] (req 18739200->18743295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.198062:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18739200->18743295] (req 18739200->18743295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.198090:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18739200->18743295] (req 18739200->18743295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.198097:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760707.198102:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18739200->18743295] (req 18739200->18743295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.198110:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18739200->18743295] (req 18739200->18743295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.198131:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18739200->18743295] (req 18739200->18743295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.198139:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076d6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18739200->18743295] (req 18739200->18743295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760707.198230:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484146560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35366us (35665us total) trans 4294971893 rc 0/0 00000100:00100000:1.0:1789760707.198240:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4604 00000100:00100000:1.0:1789760707.198249:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696450701709 time 696493778279 state 0x2 ref 0 00000001:00080000:2.0:1789760707.198318:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971893 is committed 00000100:00100000:0.0:1789760707.202890:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760707.202963:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484146688 00000100:00100000:3.0:1789760707.202972:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484146688 00000100:00100000:3.0:1789760707.203018:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696450701709 time 696493778279 state 0x0 ref 1 00000100:00100000:3.0:1789760707.203033:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696494038695 time 696493778279 state 0x1 ref 1 00000100:00100000:3.0:1789760707.203061:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696494038695 time 696533992442 state 0x0 ref 1 00000100:00100000:3.0:1789760707.203095:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484146688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760707.203135:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18743296, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:3.0:1789760707.203144:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.203159:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:3.0:1789760707.203163:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760707.203169:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18743296->18747391] (req 18743296->18747391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.203188:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18743296->18747391] (req 18743296->18747391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760707.241360:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971894 is committed 00002000:00010000:3.0:1789760707.241542:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18743296 -> 18747392 00010000:00010000:3.0:1789760707.241557:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18743296->18747391] (req 18743296->18747391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.241569:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18743296->18747391] (req 18743296->18747391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.241588:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18743296->18747391] (req 18743296->18747391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.241595:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760707.241600:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18743296->18747391] (req 18743296->18747391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.241608:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18743296->18747391] (req 18743296->18747391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.241628:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18743296->18747391] (req 18743296->18747391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.241636:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962076dd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18743296->18747391] (req 18743296->18747391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760707.242017:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484146688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38919us (39131us total) trans 4294971894 rc 0/0 00000100:00100000:3.0:1789760707.242028:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4605 00000100:00100000:3.0:1789760707.242038:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696494038695 time 696533992442 state 0x2 ref 0 00000100:00100000:3.0:1789760707.245768:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760707.245931:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484146816 00000100:00100000:2.0:1789760707.245940:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484146816 00000100:00100000:2.0:1789760707.245995:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696494038695 time 696533992442 state 0x0 ref 1 00000100:00100000:2.0:1789760707.246010:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696534252858 time 696533992442 state 0x1 ref 1 00000100:00100000:2.0:1789760707.246111:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696534252858 time 696577040946 state 0x0 ref 1 00000100:00100000:2.0:1789760707.246124:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484146816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760707.246169:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18747392, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:2.0:1789760707.246178:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.246192:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:2.0:1789760707.246196:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760707.246202:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18747392->18751487] (req 18747392->18751487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.246216:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18747392->18751487] (req 18747392->18751487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760707.280018:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971895 is committed 00002000:00010000:3.0:1789760707.280244:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18747392 -> 18751488 00010000:00010000:3.0:1789760707.280260:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18747392->18751487] (req 18747392->18751487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.280274:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18747392->18751487] (req 18747392->18751487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.280283:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18747392->18751487] (req 18747392->18751487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.280290:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760707.280295:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18747392->18751487] (req 18747392->18751487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.280303:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18747392->18751487] (req 18747392->18751487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.280323:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18747392->18751487] (req 18747392->18751487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.280336:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962076e4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18747392->18751487] (req 18747392->18751487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760707.281316:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484146816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35187us (35552us total) trans 4294971895 rc 0/0 00000100:00100000:3.0:1789760707.281328:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4606 00000100:00100000:3.0:1789760707.281337:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696534252858 time 696577040946 state 0x2 ref 0 00000100:00100000:0.0:1789760707.289921:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760707.290185:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484146944 00000100:00100000:2.0:1789760707.290195:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484146944 00000100:00100000:2.0:1789760707.290244:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696534252858 time 696577040946 state 0x0 ref 1 00000100:00100000:2.0:1789760707.290257:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696577301362 time 696577040946 state 0x1 ref 1 00000100:00100000:2.0:1789760707.290993:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696577301362 time 696621919682 state 0x0 ref 1 00000100:00100000:2.0:1789760707.291011:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484146944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760707.291059:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18751488, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:2.0:1789760707.291097:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.291113:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:2.0:1789760707.291118:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760707.291124:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18751488->18755583] (req 18751488->18755583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.291138:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18751488->18755583] (req 18751488->18755583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760707.351042:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971896 is committed 00002000:00010000:0.0:1789760707.351157:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18751488 -> 18755584 00010000:00010000:0.0:1789760707.351172:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18751488->18755583] (req 18751488->18755583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.351184:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18751488->18755583] (req 18751488->18755583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.351193:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18751488->18755583] (req 18751488->18755583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.351200:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760707.351205:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18751488->18755583] (req 18751488->18755583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.351215:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18751488->18755583] (req 18751488->18755583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.351239:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18751488->18755583] (req 18751488->18755583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.351246:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962076eb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18751488->18755583] (req 18751488->18755583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760707.351438:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484146944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60359us (61455us total) trans 4294971896 rc 0/0 00000100:00100000:0.0:1789760707.351448:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4607 00000100:00100000:0.0:1789760707.351459:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696577301362 time 696621919682 state 0x2 ref 0 00000100:00100000:2.0:1789760707.360255:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760707.360417:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484147072 00000100:00100000:1.0:1789760707.360427:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484147072 00000100:00100000:1.0:1789760707.360483:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696577301362 time 696621919682 state 0x0 ref 1 00000100:00100000:1.0:1789760707.360499:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696622180098 time 696621919682 state 0x1 ref 1 00000100:00100000:1.0:1789760707.360566:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696622180098 time 696691496606 state 0x0 ref 1 00000100:00100000:1.0:1789760707.360595:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484147072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760707.360643:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18755584, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:1.0:1789760707.360651:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.360688:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:1.0:1789760707.360693:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760707.360699:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18755584->18759679] (req 18755584->18759679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.360714:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18755584->18759679] (req 18755584->18759679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760707.401201:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971897 is committed 00002000:00010000:3.0:1789760707.401291:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18755584 -> 18759680 00010000:00010000:3.0:1789760707.401305:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18755584->18759679] (req 18755584->18759679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.401319:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18755584->18759679] (req 18755584->18759679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.401327:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18755584->18759679] (req 18755584->18759679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.401334:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760707.401339:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18755584->18759679] (req 18755584->18759679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.401348:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18755584->18759679] (req 18755584->18759679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.401470:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18755584->18759679] (req 18755584->18759679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.401478:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962076f2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18755584->18759679] (req 18755584->18759679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760707.401561:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484147072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40963us (41311us total) trans 4294971897 rc 0/0 00000100:00100000:3.0:1789760707.401578:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4608 00000100:00100000:3.0:1789760707.401648:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696622180098 time 696691496606 state 0x2 ref 0 00000100:00100000:2.0:1789760707.412790:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760707.416938:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484147200 00000100:00100000:3.0:1789760707.416952:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484147200 00000100:00100000:2.0:1789760707.420793:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696622180098 time 696691496606 state 0x0 ref 1 00000100:00100000:2.0:1789760707.420811:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696691757022 time 696691496606 state 0x1 ref 1 00000100:00100000:2.0:1789760707.420893:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696691757022 time 696751823267 state 0x0 ref 1 00000100:00100000:2.0:1789760707.420909:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484147200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760707.420957:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18759680, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:2.0:1789760707.420965:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.420980:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:2.0:1789760707.420984:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760707.420991:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18759680->18763775] (req 18759680->18763775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.421004:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18759680->18763775] (req 18759680->18763775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760707.468556:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508905856:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760707.468642:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760707.468828:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508905856 00000100:00100000:0.0:1789760707.468835:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508905856 00000100:00100000:0.0:1789760707.468886:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 413 00000100:00100000:0.0:1789760707.468895:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508905856:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:0.0:1789760707.468930:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:0.0:1789760707.468975:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508905856:12345-0@lo:41:osp-pre-0-1.0 Request processed in 81us (332us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760707.468982:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 413 00000100:00100000:0.0:1789760707.469040:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508905856:0@lo:41:osp-pre-0-1.0 00000100:00100000:2.0:1789760707.469734:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c11c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508905984:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1789760707.469770:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760707.469809:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508905984 00000100:00100000:2.0:1789760707.469814:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508905984 00000100:00100000:2.0:1789760707.469849:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 354 00000100:00100000:2.0:1789760707.469857:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508905984:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:2.0:1789760707.469871:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527499264 pending 1703936 free 3839885312 avail 3801088000 00000100:00100000:2.0:1789760707.469904:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508905984:12345-0@lo:13:osp-pre-0-0.0 Request processed in 48us (133us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760707.469911:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 354 00000100:00080000:2.0:1789760707.469926:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760707.470035:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c11c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508905984:0@lo:13:osp-pre-0-0.0 00010000:00010000:3.0:1789760707.470247:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760707.471238:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508906112:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760707.471283:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760707.471325:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508906112 00000100:00100000:0.0:1789760707.471331:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508906112 00000100:00100000:0.0:1789760707.471521:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 355 00000100:00100000:0.0:1789760707.471529:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508906112:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:0.0:1789760707.471544:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527499264 pending 1703936 free 3839885312 avail 3801088000 00000100:00100000:0.0:1789760707.472864:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508906112:12345-0@lo:13:osp-pre-0-1.0 Request processed in 1330us (1574us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760707.472876:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 355 00000100:00080000:0.0:1789760707.473435:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760707.474146:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508906112:0@lo:13:osp-pre-0-1.0 00000001:00080000:0.0:1789760707.482121:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971898 is committed 00002000:00010000:2.0:1789760707.482267:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18759680 -> 18763776 00010000:00010000:2.0:1789760707.482282:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18759680->18763775] (req 18759680->18763775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.482295:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18759680->18763775] (req 18759680->18763775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.482309:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18759680->18763775] (req 18759680->18763775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.482316:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760707.482320:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18759680->18763775] (req 18759680->18763775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.482329:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18759680->18763775] (req 18759680->18763775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.482347:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18759680->18763775] (req 18759680->18763775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.482427:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962076f9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18759680->18763775] (req 18759680->18763775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760707.482552:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484147200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61643us (69768us total) trans 4294971898 rc 0/0 00000100:00100000:2.0:1789760707.482562:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4609 00000100:00100000:2.0:1789760707.482588:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696691757022 time 696751823267 state 0x2 ref 0 00000100:00100000:0.0:1789760707.499835:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760707.499922:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484147328 00000100:00100000:3.0:1789760707.499931:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484147328 00000100:00100000:3.0:1789760707.499987:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696691757022 time 696751823267 state 0x0 ref 1 00000100:00100000:3.0:1789760707.500002:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696752083683 time 696751823267 state 0x1 ref 1 00000100:00100000:3.0:1789760707.500035:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696752083683 time 696830966321 state 0x0 ref 1 00000100:00100000:3.0:1789760707.500048:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484147328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760707.500137:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18763776, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:3.0:1789760707.500146:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.500162:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:3.0:1789760707.500166:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760707.500172:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18763776->18767871] (req 18763776->18767871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.500185:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18763776->18767871] (req 18763776->18767871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760707.547258:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971899 is committed 00002000:00010000:2.0:1789760707.547480:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18763776 -> 18767872 00010000:00010000:2.0:1789760707.547496:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18763776->18767871] (req 18763776->18767871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.547514:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18763776->18767871] (req 18763776->18767871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.547522:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18763776->18767871] (req 18763776->18767871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.547530:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760707.547534:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18763776->18767871] (req 18763776->18767871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.547543:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18763776->18767871] (req 18763776->18767871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.547562:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18763776->18767871] (req 18763776->18767871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.547639:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207700 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18763776->18767871] (req 18763776->18767871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760707.548118:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484147328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48066us (48286us total) trans 4294971899 rc 0/0 00000100:00100000:2.0:1789760707.548128:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4610 00000100:00100000:2.0:1789760707.548138:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696752083683 time 696830966321 state 0x2 ref 0 00000100:00100000:0.0:1789760707.552439:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760707.552824:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484147456 00000100:00100000:2.0:1789760707.552834:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484147456 00000100:00100000:2.0:1789760707.552886:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696752083683 time 696830966321 state 0x0 ref 1 00000100:00100000:2.0:1789760707.552903:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696831226737 time 696830966321 state 0x1 ref 1 00000100:00100000:2.0:1789760707.553007:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696831226737 time 696883936472 state 0x0 ref 1 00000100:00100000:2.0:1789760707.553021:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484147456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760707.553100:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18767872, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:2.0:1789760707.553109:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.553125:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:2.0:1789760707.553130:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760707.553136:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18767872->18771967] (req 18767872->18771967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.553151:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18767872->18771967] (req 18767872->18771967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760707.600618:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971900 is committed 00002000:00010000:1.0:1789760707.600894:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18767872 -> 18771968 00010000:00010000:1.0:1789760707.600910:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18767872->18771967] (req 18767872->18771967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.600922:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18767872->18771967] (req 18767872->18771967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.600932:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18767872->18771967] (req 18767872->18771967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.600939:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760707.600944:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18767872->18771967] (req 18767872->18771967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.600952:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18767872->18771967] (req 18767872->18771967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.600970:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18767872->18771967] (req 18767872->18771967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760707.600978:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207707 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18767872->18771967] (req 18767872->18771967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760707.601102:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484147456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48047us (48635us total) trans 4294971900 rc 0/0 00000100:00100000:1.0:1789760707.601112:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4611 00000100:00100000:1.0:1789760707.601121:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696831226737 time 696883936472 state 0x2 ref 0 00000100:00100000:1.0:1789760707.605902:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760707.605973:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484147584 00000100:00100000:3.0:1789760707.605982:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484147584 00000100:00100000:3.0:1789760707.606031:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696831226737 time 696883936472 state 0x0 ref 1 00000100:00100000:3.0:1789760707.606046:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696884196888 time 696883936472 state 0x1 ref 1 00000100:00100000:3.0:1789760707.606105:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696884196888 time 696937036188 state 0x0 ref 1 00000100:00100000:3.0:1789760707.606118:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484147584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760707.606166:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18771968, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:3.0:1789760707.606176:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.606191:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=696s v=(0 0) 00010000:00010000:3.0:1789760707.606195:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760707.606202:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18771968->18776063] (req 18771968->18776063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.606220:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18771968->18776063] (req 18771968->18776063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760707.680796:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971901 is committed 00002000:00010000:3.0:1789760707.681617:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18771968 -> 18776064 00010000:00010000:3.0:1789760707.681632:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18771968->18776063] (req 18771968->18776063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.681644:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18771968->18776063] (req 18771968->18776063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.681663:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18771968->18776063] (req 18771968->18776063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.681672:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760707.681932:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18771968->18776063] (req 18771968->18776063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.681942:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18771968->18776063] (req 18771968->18776063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.681963:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18771968->18776063] (req 18771968->18776063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.681970:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620770e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18771968->18776063] (req 18771968->18776063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760707.682061:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484147584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 75941us (76165us total) trans 4294971901 rc 0/0 00000100:00100000:3.0:1789760707.682088:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4612 00000100:00100000:3.0:1789760707.682098:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696884196888 time 696937036188 state 0x2 ref 0 00000100:00100000:2.0:1789760707.688252:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760707.688335:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484147712 00000100:00100000:0.0:1789760707.688345:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484147712 00000100:00100000:0.0:1789760707.688408:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696884196888 time 696937036188 state 0x0 ref 1 00000100:00100000:0.0:1789760707.688423:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696937296604 time 696937036188 state 0x1 ref 1 00000100:00100000:0.0:1789760707.688454:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696937296604 time 697019385213 state 0x0 ref 1 00000100:00100000:0.0:1789760707.688468:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484147712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760707.688524:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18776064, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:0.0:1789760707.688534:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.688547:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:0.0:1789760707.688552:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760707.688558:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18776064->18780159] (req 18776064->18780159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760707.688587:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18776064->18780159] (req 18776064->18780159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760707.776629:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971902 is committed 00002000:00010000:3.0:1789760707.777280:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18776064 -> 18780160 00010000:00010000:3.0:1789760707.777296:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18776064->18780159] (req 18776064->18780159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.777308:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18776064->18780159] (req 18776064->18780159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.777317:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18776064->18780159] (req 18776064->18780159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.777325:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760707.777330:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18776064->18780159] (req 18776064->18780159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.777339:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18776064->18780159] (req 18776064->18780159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.777513:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18776064->18780159] (req 18776064->18780159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.777526:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207715 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18776064->18780159] (req 18776064->18780159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760707.777818:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484147712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 89349us (89571us total) trans 4294971902 rc 0/0 00000100:00100000:3.0:1789760707.777829:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4613 00000100:00100000:3.0:1789760707.777844:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696937296604 time 697019385213 state 0x2 ref 0 00000100:00100000:0.0:1789760707.780527:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760707.780704:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484147840 00000100:00100000:2.0:1789760707.780714:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484147840 00000100:00100000:2.0:1789760707.780765:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 696937296604 time 697019385213 state 0x0 ref 1 00000100:00100000:2.0:1789760707.780780:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697019645629 time 697019385213 state 0x1 ref 1 00000100:00100000:2.0:1789760707.780811:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697019645629 time 697111742581 state 0x0 ref 1 00000100:00100000:2.0:1789760707.780822:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484147840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760707.780868:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18780160, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:2.0:1789760707.780877:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.780893:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:2.0:1789760707.780897:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760707.780903:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18780160->18784255] (req 18780160->18784255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.780918:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18780160->18784255] (req 18780160->18784255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760707.833020:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971903 is committed 00002000:00010000:2.0:1789760707.833663:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18780160 -> 18784256 00010000:00010000:2.0:1789760707.833687:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18780160->18784255] (req 18780160->18784255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.833701:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18780160->18784255] (req 18780160->18784255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.833711:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18780160->18784255] (req 18780160->18784255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.833718:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760707.833723:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18780160->18784255] (req 18780160->18784255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.833734:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18780160->18784255] (req 18780160->18784255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.833753:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18780160->18784255] (req 18780160->18784255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.833762:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620771c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18780160->18784255] (req 18780160->18784255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760707.833971:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484147840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53144us (53447us total) trans 4294971903 rc 0/0 00000100:00100000:2.0:1789760707.833981:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4614 00000100:00100000:2.0:1789760707.833990:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697019645629 time 697111742581 state 0x2 ref 0 00000100:00100000:1.0:1789760707.843241:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760707.843585:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484147968 00000100:00100000:2.0:1789760707.843595:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484147968 00000100:00100000:2.0:1789760707.843652:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697019645629 time 697111742581 state 0x0 ref 1 00000100:00100000:2.0:1789760707.843685:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697112002997 time 697111742581 state 0x1 ref 1 00000100:00100000:2.0:1789760707.843712:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697112002997 time 697174643513 state 0x0 ref 1 00000100:00100000:2.0:1789760707.843725:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484147968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760707.843772:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18784256, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:2.0:1789760707.843780:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.843794:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:2.0:1789760707.843798:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760707.843805:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18784256->18788351] (req 18784256->18788351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760707.843827:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18784256->18788351] (req 18784256->18788351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760707.911283:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18784256 -> 18788352 00010000:00010000:3.0:1789760707.911304:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18784256->18788351] (req 18784256->18788351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.911318:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18784256->18788351] (req 18784256->18788351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.911329:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18784256->18788351] (req 18784256->18788351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.911337:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760707.911341:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18784256->18788351] (req 18784256->18788351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.911361:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18784256->18788351] (req 18784256->18788351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.911385:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18784256->18788351] (req 18784256->18788351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.911393:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207723 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18784256->18788351] (req 18784256->18788351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760707.911481:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484147968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67754us (68246us total) trans 4294971904 rc 0/0 00000100:00100000:3.0:1789760707.911492:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4615 00000100:00100000:3.0:1789760707.911502:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697112002997 time 697174643513 state 0x2 ref 0 00000001:00080000:0.0:1789760707.912033:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971904 is committed 00000100:00100000:0.0:1789760707.918708:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760707.918802:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484148096 00000100:00100000:3.0:1789760707.918811:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484148096 00000100:00100000:3.0:1789760707.918862:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697112002997 time 697174643513 state 0x0 ref 1 00000100:00100000:3.0:1789760707.918878:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697174903929 time 697174643513 state 0x1 ref 1 00000100:00100000:3.0:1789760707.918910:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697174903929 time 697249841453 state 0x0 ref 1 00000100:00100000:3.0:1789760707.918922:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484148096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760707.918966:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18788352, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:3.0:1789760707.918974:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.918990:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:3.0:1789760707.918995:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760707.919001:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18788352->18792447] (req 18788352->18792447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.919015:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18788352->18792447] (req 18788352->18792447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760707.981041:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760707.981189:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508906240:0@lo:41:osp-pre-1-0.0 00000100:00100000:0.0:1789760707.981264:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760707.993635:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508906240 00000100:00100000:1.0:1789760707.993649:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508906240 00000100:00100000:1.0:1789760707.993731:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 414 00000100:00100000:1.0:1789760707.993740:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508906240:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760707.993780:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760707.993881:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508906240:12345-0@lo:41:osp-pre-1-0.0 Request processed in 142us (12617us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760707.993887:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 414 00000100:00100000:2.0:1789760707.994172:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508906240:0@lo:41:osp-pre-1-0.0 00000001:00080000:1.0:1789760707.999027:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971905 is committed 00002000:00010000:3.0:1789760707.999153:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18788352 -> 18792448 00010000:00010000:3.0:1789760707.999167:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18788352->18792447] (req 18788352->18792447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.999179:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18788352->18792447] (req 18788352->18792447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.999188:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18788352->18792447] (req 18788352->18792447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.999195:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760707.999200:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18788352->18792447] (req 18788352->18792447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.999209:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18788352->18792447] (req 18788352->18792447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.999227:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18788352->18792447] (req 18788352->18792447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760707.999235:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620772a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18788352->18792447] (req 18788352->18792447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760707.999313:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484148096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 80388us (80609us total) trans 4294971905 rc 0/0 00000100:00100000:3.0:1789760707.999324:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4616 00000100:00100000:3.0:1789760707.999334:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697174903929 time 697249841453 state 0x2 ref 0 00000100:00100000:0.0:1789760708.003058:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760708.003222:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484148224 00000100:00100000:3.0:1789760708.003233:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484148224 00000100:00100000:3.0:1789760708.003301:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697174903929 time 697249841453 state 0x0 ref 1 00000100:00100000:3.0:1789760708.003316:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697250101869 time 697249841453 state 0x1 ref 1 00000100:00100000:3.0:1789760708.003348:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697250101869 time 697334279282 state 0x0 ref 1 00000100:00100000:3.0:1789760708.003372:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484148224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760708.003417:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18792448, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:3.0:1789760708.003427:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.003441:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:3.0:1789760708.003446:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760708.003452:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18792448->18796543] (req 18792448->18796543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.003466:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18792448->18796543] (req 18792448->18796543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1789760708.003930:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:1.0:1789760708.056452:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971906 is committed 00002000:00010000:3.0:1789760708.056636:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18792448 -> 18796544 00010000:00010000:3.0:1789760708.056661:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18792448->18796543] (req 18792448->18796543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.056675:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18792448->18796543] (req 18792448->18796543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.056683:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18792448->18796543] (req 18792448->18796543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.056690:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760708.056694:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18792448->18796543] (req 18792448->18796543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.056703:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18792448->18796543] (req 18792448->18796543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.056722:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18792448->18796543] (req 18792448->18796543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.056730:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207731 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18792448->18796543] (req 18792448->18796543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760708.056941:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484148224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53568us (53889us total) trans 4294971906 rc 0/0 00000100:00100000:3.0:1789760708.056955:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4617 00000100:00100000:3.0:1789760708.056964:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697250101869 time 697334279282 state 0x2 ref 0 00000100:00100000:1.0:1789760708.061169:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760708.061278:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484148352 00000100:00100000:2.0:1789760708.061287:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484148352 00000100:00100000:2.0:1789760708.061341:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697250101869 time 697334279282 state 0x0 ref 1 00000100:00100000:2.0:1789760708.061366:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697334539698 time 697334279282 state 0x1 ref 1 00000100:00100000:2.0:1789760708.061423:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697334539698 time 697392353407 state 0x0 ref 1 00000100:00100000:2.0:1789760708.061436:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484148352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760708.061484:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18796544, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:2.0:1789760708.061493:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.061515:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:2.0:1789760708.061520:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760708.061526:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18796544->18800639] (req 18796544->18800639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.061540:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18796544->18800639] (req 18796544->18800639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760708.139212:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971907 is committed 00002000:00010000:0.0:1789760708.139503:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18796544 -> 18800640 00010000:00010000:0.0:1789760708.139518:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18796544->18800639] (req 18796544->18800639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.139531:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18796544->18800639] (req 18796544->18800639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.139540:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18796544->18800639] (req 18796544->18800639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.139548:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760708.139553:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18796544->18800639] (req 18796544->18800639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.139596:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18796544->18800639] (req 18796544->18800639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.139617:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18796544->18800639] (req 18796544->18800639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.139625:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207738 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18796544->18800639] (req 18796544->18800639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760708.140318:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484148352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 78877us (79151us total) trans 4294971907 rc 0/0 00000100:00100000:0.0:1789760708.140329:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4618 00000100:00100000:0.0:1789760708.140339:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697334539698 time 697392353407 state 0x2 ref 0 00000100:00100000:0.0:1789760708.144135:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760708.144223:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484148480 00000100:00100000:3.0:1789760708.144233:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484148480 00000100:00100000:3.0:1789760708.144297:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697334539698 time 697392353407 state 0x0 ref 1 00000100:00100000:3.0:1789760708.144313:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697392613823 time 697392353407 state 0x1 ref 1 00000100:00100000:3.0:1789760708.144343:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697392613823 time 697475273847 state 0x0 ref 1 00000100:00100000:3.0:1789760708.144363:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484148480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760708.144412:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18800640, mtime: 0x6aad94c3, atime: 0x0, ctime: 0x6aad94c3, blocks: 0x9007 00010000:00010000:3.0:1789760708.144421:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.144436:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:3.0:1789760708.144440:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760708.144446:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18800640->18804735] (req 18800640->18804735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.144460:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18800640->18804735] (req 18800640->18804735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760708.178026:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971908 is committed 00002000:00010000:2.0:1789760708.178254:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18800640 -> 18804736 00002000:00010000:2.0:1789760708.178259:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760707 -> 1789760708 00002000:00010000:2.0:1789760708.178262:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760707 -> 1789760708 00010000:00010000:2.0:1789760708.178274:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18800640->18804735] (req 18800640->18804735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.178287:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18800640->18804735] (req 18800640->18804735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.178296:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18800640->18804735] (req 18800640->18804735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.178303:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760708.178308:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18800640->18804735] (req 18800640->18804735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.178318:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18800640->18804735] (req 18800640->18804735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.178344:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18800640->18804735] (req 18800640->18804735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.178364:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620773f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18800640->18804735] (req 18800640->18804735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760708.178455:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484148480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34091us (34324us total) trans 4294971908 rc 0/0 00000100:00100000:2.0:1789760708.178467:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4619 00000100:00100000:2.0:1789760708.178477:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697392613823 time 697475273847 state 0x2 ref 0 00000100:00100000:0.0:1789760708.182166:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760708.182390:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484148608 00000100:00100000:3.0:1789760708.182401:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484148608 00000100:00100000:3.0:1789760708.182454:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697392613823 time 697475273847 state 0x0 ref 1 00000100:00100000:3.0:1789760708.182469:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697475534263 time 697475273847 state 0x1 ref 1 00000100:00100000:3.0:1789760708.182530:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697475534263 time 697513459990 state 0x0 ref 1 00000100:00100000:3.0:1789760708.182543:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484148608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760708.182608:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18804736, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:3.0:1789760708.182617:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.182633:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:3.0:1789760708.182637:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760708.182644:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18804736->18808831] (req 18804736->18808831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.182762:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18804736->18808831] (req 18804736->18808831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760708.235633:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18804736 -> 18808832 00010000:00010000:1.0:1789760708.235660:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18804736->18808831] (req 18804736->18808831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.235673:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18804736->18808831] (req 18804736->18808831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.235682:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18804736->18808831] (req 18804736->18808831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.235690:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760708.235695:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18804736->18808831] (req 18804736->18808831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.235705:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18804736->18808831] (req 18804736->18808831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.235725:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18804736->18808831] (req 18804736->18808831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.235733:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207746 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18804736->18808831] (req 18804736->18808831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760708.235827:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484148608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53282us (53666us total) trans 4294971909 rc 0/0 00000100:00100000:1.0:1789760708.235838:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4620 00000100:00100000:1.0:1789760708.235847:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697475534263 time 697513459990 state 0x2 ref 0 00000001:00080000:1.0:1789760708.235889:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971909 is committed 00000100:00100000:1.0:1789760708.240400:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760708.240767:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484148736 00000100:00100000:0.0:1789760708.240778:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484148736 00000100:00100000:0.0:1789760708.240840:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697475534263 time 697513459990 state 0x0 ref 1 00000100:00100000:0.0:1789760708.240857:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697513720406 time 697513459990 state 0x1 ref 1 00000100:00100000:0.0:1789760708.240893:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697513720406 time 697571823152 state 0x0 ref 1 00000100:00100000:0.0:1789760708.240907:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484148736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760708.240959:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18808832, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:0.0:1789760708.240968:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.240983:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:0.0:1789760708.240987:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760708.240994:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18808832->18812927] (req 18808832->18812927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.241008:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18808832->18812927] (req 18808832->18812927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760708.297551:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971910 is committed 00002000:00010000:3.0:1789760708.297830:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18808832 -> 18812928 00010000:00010000:3.0:1789760708.297844:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18808832->18812927] (req 18808832->18812927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.297857:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18808832->18812927] (req 18808832->18812927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.297867:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18808832->18812927] (req 18808832->18812927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.297875:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760708.297880:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18808832->18812927] (req 18808832->18812927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.297889:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18808832->18812927] (req 18808832->18812927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.297909:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18808832->18812927] (req 18808832->18812927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.297916:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620774d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18808832->18812927] (req 18808832->18812927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760708.298010:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484148736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57102us (57615us total) trans 4294971910 rc 0/0 00000100:00100000:3.0:1789760708.298021:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4621 00000100:00100000:3.0:1789760708.298030:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697513720406 time 697571823152 state 0x2 ref 0 00000100:00100000:2.0:1789760708.301938:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760708.302033:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484148864 00000100:00100000:0.0:1789760708.302043:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484148864 00000100:00100000:0.0:1789760708.302146:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697513720406 time 697571823152 state 0x0 ref 1 00000100:00100000:0.0:1789760708.302160:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697572083568 time 697571823152 state 0x1 ref 1 00000100:00100000:0.0:1789760708.302180:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697572083568 time 697633111339 state 0x0 ref 1 00000100:00100000:0.0:1789760708.302192:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484148864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760708.302237:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18812928, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:0.0:1789760708.302247:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.302264:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:0.0:1789760708.302268:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760708.302275:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18812928->18817023] (req 18812928->18817023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.302289:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18812928->18817023] (req 18812928->18817023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760708.409019:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18812928 -> 18817024 00010000:00010000:0.0:1789760708.409039:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18812928->18817023] (req 18812928->18817023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.409051:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18812928->18817023] (req 18812928->18817023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.409059:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18812928->18817023] (req 18812928->18817023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.409087:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760708.409091:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18812928->18817023] (req 18812928->18817023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.409100:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18812928->18817023] (req 18812928->18817023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.409120:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18812928->18817023] (req 18812928->18817023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.409134:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207754 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18812928->18817023] (req 18812928->18817023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760708.409239:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484148864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 107045us (107305us total) trans 4294971911 rc 0/0 00000100:00100000:0.0:1789760708.409249:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4622 00000100:00100000:0.0:1789760708.409259:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697572083568 time 697633111339 state 0x2 ref 0 00000100:00100000:0.0:1789760708.419369:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760708.419486:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484148992 00000100:00100000:3.0:1789760708.419495:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484148992 00000100:00100000:3.0:1789760708.419547:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697572083568 time 697633111339 state 0x0 ref 1 00000100:00100000:3.0:1789760708.419573:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697633371755 time 697633111339 state 0x1 ref 1 00000100:00100000:3.0:1789760708.419604:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697633371755 time 697750534994 state 0x0 ref 1 00000100:00100000:3.0:1789760708.419621:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484148992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760708.419679:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18817024, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:3.0:1789760708.419688:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.419704:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:3.0:1789760708.419709:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760708.419716:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18817024->18821119] (req 18817024->18821119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.419730:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18817024->18821119] (req 18817024->18821119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760708.423974:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971911 is committed 00000001:00080000:3.0:1789760708.473279:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971912 is committed 00002000:00010000:1.0:1789760708.473446:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18817024 -> 18821120 00010000:00010000:1.0:1789760708.473461:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18817024->18821119] (req 18817024->18821119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.473473:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18817024->18821119] (req 18817024->18821119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.473481:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18817024->18821119] (req 18817024->18821119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.473487:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760708.473492:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18817024->18821119] (req 18817024->18821119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.473501:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18817024->18821119] (req 18817024->18821119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.473519:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18817024->18821119] (req 18817024->18821119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.473526:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620775b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18817024->18821119] (req 18817024->18821119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760708.473762:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484148992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54135us (54405us total) trans 4294971912 rc 0/0 00000100:00100000:1.0:1789760708.473773:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4623 00000100:00100000:1.0:1789760708.473783:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697633371755 time 697750534994 state 0x2 ref 0 00000100:00100000:1.0:1789760708.479328:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760708.481571:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484149120 00000100:00100000:3.0:1789760708.481584:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484149120 00000100:00100000:3.0:1789760708.481645:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697633371755 time 697750534994 state 0x0 ref 1 00000100:00100000:3.0:1789760708.481661:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697750795410 time 697750534994 state 0x1 ref 1 00000100:00100000:3.0:1789760708.481694:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697750795410 time 697812624882 state 0x0 ref 1 00000100:00100000:3.0:1789760708.481708:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484149120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760708.481752:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18821120, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:3.0:1789760708.481760:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.481777:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:3.0:1789760708.481781:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760708.481787:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18821120->18825215] (req 18821120->18825215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.481801:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18821120->18825215] (req 18821120->18825215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.492959:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760708.536083:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971913 is committed 00002000:00010000:3.0:1789760708.536936:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18821120 -> 18825216 00010000:00010000:3.0:1789760708.536964:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18821120->18825215] (req 18821120->18825215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.536982:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18821120->18825215] (req 18821120->18825215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.536991:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18821120->18825215] (req 18821120->18825215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.536998:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760708.537003:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18821120->18825215] (req 18821120->18825215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.537011:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18821120->18825215] (req 18821120->18825215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.537029:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18821120->18825215] (req 18821120->18825215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.537038:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207762 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18821120->18825215] (req 18821120->18825215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760708.537166:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484149120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55456us (57842us total) trans 4294971913 rc 0/0 00000100:00100000:3.0:1789760708.537175:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4624 00000100:00100000:3.0:1789760708.537184:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697750795410 time 697812624882 state 0x2 ref 0 00000100:00100000:0.0:1789760708.539930:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760708.540020:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484149248 00000100:00100000:3.0:1789760708.540030:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484149248 00000100:00100000:3.0:1789760708.540101:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697750795410 time 697812624882 state 0x0 ref 1 00000100:00100000:3.0:1789760708.540116:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697812885298 time 697812624882 state 0x1 ref 1 00000100:00100000:3.0:1789760708.540147:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697812885298 time 697871077743 state 0x0 ref 1 00000100:00100000:3.0:1789760708.540159:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484149248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760708.540203:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18825216, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:3.0:1789760708.540212:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.540228:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:3.0:1789760708.540232:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760708.540238:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18825216->18829311] (req 18825216->18829311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.540254:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18825216->18829311] (req 18825216->18829311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760708.597603:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18825216 -> 18829312 00010000:00010000:3.0:1789760708.597622:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18825216->18829311] (req 18825216->18829311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.597636:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18825216->18829311] (req 18825216->18829311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.597645:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18825216->18829311] (req 18825216->18829311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.597718:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760708.597724:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18825216->18829311] (req 18825216->18829311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.597734:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18825216->18829311] (req 18825216->18829311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.597754:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18825216->18829311] (req 18825216->18829311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.597762:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207769 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18825216->18829311] (req 18825216->18829311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760708.597964:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484149248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57802us (58036us total) trans 4294971914 rc 0/0 00000100:00100000:3.0:1789760708.597974:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4625 00000100:00100000:3.0:1789760708.597985:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697812885298 time 697871077743 state 0x2 ref 0 00000001:00080000:1.0:1789760708.598044:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971914 is committed 00000100:00100000:0.0:1789760708.602801:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760708.602946:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484149376 00000100:00100000:2.0:1789760708.602955:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484149376 00000100:00100000:2.0:1789760708.603008:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697812885298 time 697871077743 state 0x0 ref 1 00000100:00100000:2.0:1789760708.603023:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697871338159 time 697871077743 state 0x1 ref 1 00000100:00100000:2.0:1789760708.603052:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697871338159 time 697933983353 state 0x0 ref 1 00000100:00100000:2.0:1789760708.603097:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484149376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760708.603143:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18829312, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:2.0:1789760708.603153:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.603174:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:2.0:1789760708.603178:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760708.603184:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18829312->18833407] (req 18829312->18833407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760708.603198:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18829312->18833407] (req 18829312->18833407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760708.652965:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18829312 -> 18833408 00010000:00010000:0.0:1789760708.652986:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18829312->18833407] (req 18829312->18833407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.653001:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18829312->18833407] (req 18829312->18833407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.653010:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18829312->18833407] (req 18829312->18833407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.653017:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760708.653022:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18829312->18833407] (req 18829312->18833407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.653038:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18829312->18833407] (req 18829312->18833407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.653057:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18829312->18833407] (req 18829312->18833407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.653085:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207770 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18829312->18833407] (req 18829312->18833407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760708.653179:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484149376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50111us (50383us total) trans 4294971915 rc 0/0 00000100:00100000:0.0:1789760708.653190:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4626 00000100:00100000:0.0:1789760708.653200:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697871338159 time 697933983353 state 0x2 ref 0 00000001:00080000:1.0:1789760708.653969:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971915 is committed 00000100:00100000:2.0:1789760708.659877:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760708.659969:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484149504 00000100:00100000:3.0:1789760708.659978:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484149504 00000100:00100000:3.0:1789760708.660033:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697871338159 time 697933983353 state 0x0 ref 1 00000100:00100000:3.0:1789760708.660048:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697934243769 time 697933983353 state 0x1 ref 1 00000100:00100000:3.0:1789760708.660108:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697934243769 time 697991038465 state 0x0 ref 1 00000100:00100000:3.0:1789760708.660120:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484149504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760708.660168:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18833408, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:3.0:1789760708.660177:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.660192:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=697s v=(0 0) 00010000:00010000:3.0:1789760708.660196:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760708.660202:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18833408->18837503] (req 18833408->18837503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.660215:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18833408->18837503] (req 18833408->18837503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760708.692027:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18833408 -> 18837504 00010000:00010000:3.0:1789760708.692048:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18833408->18837503] (req 18833408->18837503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.692062:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18833408->18837503] (req 18833408->18837503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.692101:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18833408->18837503] (req 18833408->18837503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.692108:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760708.692114:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18833408->18837503] (req 18833408->18837503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.692123:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18833408->18837503] (req 18833408->18837503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.692145:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18833408->18837503] (req 18833408->18837503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.692153:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207777 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18833408->18837503] (req 18833408->18837503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760708.692236:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484149504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32113us (32365us total) trans 4294971916 rc 0/0 00000100:00100000:3.0:1789760708.692250:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4627 00000100:00100000:3.0:1789760708.692259:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697934243769 time 697991038465 state 0x2 ref 0 00000001:00080000:0.0:1789760708.693679:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971916 is committed 00000100:00100000:3.0:1789760708.697011:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760708.697110:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484149632 00000100:00100000:1.0:1789760708.697120:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484149632 00000100:00100000:1.0:1789760708.697183:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697934243769 time 697991038465 state 0x0 ref 1 00000100:00100000:1.0:1789760708.697198:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697991298881 time 697991038465 state 0x1 ref 1 00000100:00100000:1.0:1789760708.697226:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697991298881 time 698028157440 state 0x0 ref 1 00000100:00100000:1.0:1789760708.697238:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484149632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760708.697287:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18837504, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:1.0:1789760708.697295:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.697311:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:1.0:1789760708.697315:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760708.697321:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18837504->18841599] (req 18837504->18841599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.697334:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18837504->18841599] (req 18837504->18841599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760708.722001:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971917 is committed 00002000:00010000:1.0:1789760708.722218:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18837504 -> 18841600 00010000:00010000:1.0:1789760708.722232:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18837504->18841599] (req 18837504->18841599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.722245:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18837504->18841599] (req 18837504->18841599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.722254:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18837504->18841599] (req 18837504->18841599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.722261:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760708.722266:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18837504->18841599] (req 18837504->18841599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.722275:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18837504->18841599] (req 18837504->18841599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.722292:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18837504->18841599] (req 18837504->18841599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.722300:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620777e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18837504->18841599] (req 18837504->18841599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760708.722398:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484149632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25158us (25392us total) trans 4294971917 rc 0/0 00000100:00100000:1.0:1789760708.722409:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4628 00000100:00100000:1.0:1789760708.722418:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697991298881 time 698028157440 state 0x2 ref 0 00000100:00100000:0.0:1789760708.726419:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760708.726541:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484149760 00000100:00100000:3.0:1789760708.726551:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484149760 00000100:00100000:3.0:1789760708.726611:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 697991298881 time 698028157440 state 0x0 ref 1 00000100:00100000:3.0:1789760708.726625:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698028417856 time 698028157440 state 0x1 ref 1 00000100:00100000:1.0:1789760708.726728:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698028417856 time 698057656555 state 0x0 ref 1 00000100:00100000:1.0:1789760708.726743:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484149760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760708.726789:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18841600, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:1.0:1789760708.726797:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.726812:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:1.0:1789760708.726816:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760708.726822:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18841600->18845695] (req 18841600->18845695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.726835:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18841600->18845695] (req 18841600->18845695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760708.797916:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971918 is committed 00002000:00010000:3.0:1789760708.803536:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18841600 -> 18845696 00010000:00010000:3.0:1789760708.803567:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18841600->18845695] (req 18841600->18845695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.803581:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18841600->18845695] (req 18841600->18845695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.803591:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18841600->18845695] (req 18841600->18845695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.803599:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760708.803603:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18841600->18845695] (req 18841600->18845695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.803612:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18841600->18845695] (req 18841600->18845695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.803864:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18841600->18845695] (req 18841600->18845695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.803871:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207785 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18841600->18845695] (req 18841600->18845695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760708.806311:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484149760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79561us (79892us total) trans 4294971918 rc 0/0 00000100:00100000:2.0:1789760708.806325:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4629 00000100:00100000:2.0:1789760708.806335:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698028417856 time 698057656555 state 0x2 ref 0 00000100:00100000:2.0:1789760708.811293:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760708.811900:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484149888 00000100:00100000:3.0:1789760708.811910:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484149888 00000100:00100000:3.0:1789760708.811963:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698028417856 time 698057656555 state 0x0 ref 1 00000100:00100000:3.0:1789760708.811977:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698057916971 time 698057656555 state 0x1 ref 1 00000100:00100000:3.0:1789760708.812008:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698057916971 time 698142939196 state 0x0 ref 1 00000100:00100000:3.0:1789760708.812021:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484149888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760708.812104:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18845696, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:3.0:1789760708.812113:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.812130:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:3.0:1789760708.812134:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760708.812141:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18845696->18849791] (req 18845696->18849791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.812156:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18845696->18849791] (req 18845696->18849791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760708.888185:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971919 is committed 00002000:00010000:3.0:1789760708.888321:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18845696 -> 18849792 00010000:00010000:3.0:1789760708.888337:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18845696->18849791] (req 18845696->18849791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.888363:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18845696->18849791] (req 18845696->18849791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.888373:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18845696->18849791] (req 18845696->18849791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.888379:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760708.888384:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18845696->18849791] (req 18845696->18849791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.888393:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18845696->18849791] (req 18845696->18849791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.888413:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18845696->18849791] (req 18845696->18849791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760708.888421:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620778c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18845696->18849791] (req 18845696->18849791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760708.888520:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484149888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 76496us (77231us total) trans 4294971919 rc 0/0 00000100:00100000:3.0:1789760708.888533:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4630 00000100:00100000:3.0:1789760708.888543:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698057916971 time 698142939196 state 0x2 ref 0 00000100:00100000:3.0:1789760708.897665:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760708.897764:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484150016 00000100:00100000:0.0:1789760708.897775:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484150016 00000100:00100000:0.0:1789760708.897833:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698057916971 time 698142939196 state 0x0 ref 1 00000100:00100000:0.0:1789760708.897849:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698143199612 time 698142939196 state 0x1 ref 1 00000100:00100000:0.0:1789760708.897880:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698143199612 time 698228811686 state 0x0 ref 1 00000100:00100000:0.0:1789760708.897896:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484150016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760708.897947:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18849792, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:0.0:1789760708.897955:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.897970:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:0.0:1789760708.897974:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760708.897981:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18849792->18853887] (req 18849792->18853887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760708.897996:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18849792->18853887] (req 18849792->18853887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760708.961821:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971920 is committed 00002000:00010000:1.0:1789760708.962113:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18849792 -> 18853888 00010000:00010000:1.0:1789760708.962128:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18849792->18853887] (req 18849792->18853887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.962141:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18849792->18853887] (req 18849792->18853887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.962149:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18849792->18853887] (req 18849792->18853887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.962157:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760708.962162:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18849792->18853887] (req 18849792->18853887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.962172:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18849792->18853887] (req 18849792->18853887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.962192:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18849792->18853887] (req 18849792->18853887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.962199:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207793 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18849792->18853887] (req 18849792->18853887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760708.962292:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484150016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 64396us (64631us total) trans 4294971920 rc 0/0 00000100:00100000:1.0:1789760708.962302:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4631 00000100:00100000:1.0:1789760708.962311:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698143199612 time 698228811686 state 0x2 ref 0 00000100:00100000:0.0:1789760708.966498:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760708.968372:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484150144 00000100:00100000:1.0:1789760708.968387:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484150144 00000100:00100000:1.0:1789760708.968462:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698143199612 time 698228811686 state 0x0 ref 1 00000100:00100000:1.0:1789760708.968478:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698229072102 time 698228811686 state 0x1 ref 1 00000100:00100000:1.0:1789760708.968517:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698229072102 time 698299448370 state 0x0 ref 1 00000100:00100000:1.0:1789760708.968531:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484150144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760708.968590:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18853888, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:1.0:1789760708.968599:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.968615:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:1.0:1789760708.968620:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760708.968626:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18853888->18857983] (req 18853888->18857983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760708.968640:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18853888->18857983] (req 18853888->18857983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760709.027050:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971921 is committed 00002000:00010000:2.0:1789760709.027224:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18853888 -> 18857984 00010000:00010000:2.0:1789760709.027241:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18853888->18857983] (req 18853888->18857983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.027255:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18853888->18857983] (req 18853888->18857983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.027263:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18853888->18857983] (req 18853888->18857983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.027270:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760709.027276:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18853888->18857983] (req 18853888->18857983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.027285:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18853888->18857983] (req 18853888->18857983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.027306:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18853888->18857983] (req 18853888->18857983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.027314:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620779a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18853888->18857983] (req 18853888->18857983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760709.027677:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484150144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59142us (61181us total) trans 4294971921 rc 0/0 00000100:00100000:2.0:1789760709.027688:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4632 00000100:00100000:2.0:1789760709.027698:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698229072102 time 698299448370 state 0x2 ref 0 00000100:00100000:2.0:1789760709.034614:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760709.034767:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484150272 00000100:00100000:1.0:1789760709.034778:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484150272 00000100:00100000:1.0:1789760709.034838:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698229072102 time 698299448370 state 0x0 ref 1 00000100:00100000:1.0:1789760709.034854:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698299708786 time 698299448370 state 0x1 ref 1 00000100:00100000:1.0:1789760709.034883:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698299708786 time 698365814590 state 0x0 ref 1 00000100:00100000:1.0:1789760709.034896:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484150272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760709.035060:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18857984, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:1.0:1789760709.035105:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.035217:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:1.0:1789760709.035221:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760709.035228:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18857984->18862079] (req 18857984->18862079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.035244:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18857984->18862079] (req 18857984->18862079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760709.088822:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971922 is committed 00002000:00010000:3.0:1789760709.089175:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18857984 -> 18862080 00010000:00010000:3.0:1789760709.089192:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18857984->18862079] (req 18857984->18862079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.089204:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18857984->18862079] (req 18857984->18862079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.089212:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18857984->18862079] (req 18857984->18862079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.089219:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760709.089224:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18857984->18862079] (req 18857984->18862079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.089232:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18857984->18862079] (req 18857984->18862079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.089250:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18857984->18862079] (req 18857984->18862079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.089257:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962077a1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18857984->18862079] (req 18857984->18862079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760709.089408:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484150272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54447us (54736us total) trans 4294971922 rc 0/0 00000100:00100000:3.0:1789760709.089419:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4633 00000100:00100000:3.0:1789760709.089429:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698299708786 time 698365814590 state 0x2 ref 0 00000100:00100000:1.0:1789760709.093377:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760709.093483:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484150400 00000100:00100000:3.0:1789760709.093493:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484150400 00000100:00100000:3.0:1789760709.093542:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698299708786 time 698365814590 state 0x0 ref 1 00000100:00100000:3.0:1789760709.093565:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698366075006 time 698365814590 state 0x1 ref 1 00000100:00100000:3.0:1789760709.093592:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698366075006 time 698424523667 state 0x0 ref 1 00000100:00100000:3.0:1789760709.093604:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484150400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760709.093660:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18862080, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:3.0:1789760709.093669:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.093683:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:3.0:1789760709.093688:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760709.093694:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18862080->18866175] (req 18862080->18866175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.093708:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18862080->18866175] (req 18862080->18866175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760709.141331:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971923 is committed 00002000:00010000:3.0:1789760709.141504:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18862080 -> 18866176 00010000:00010000:3.0:1789760709.141522:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18862080->18866175] (req 18862080->18866175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.141536:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18862080->18866175] (req 18862080->18866175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.141545:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18862080->18866175] (req 18862080->18866175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.141552:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760709.141566:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18862080->18866175] (req 18862080->18866175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.141576:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18862080->18866175] (req 18862080->18866175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.141597:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18862080->18866175] (req 18862080->18866175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.141605:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077a8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18862080->18866175] (req 18862080->18866175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760709.141765:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484150400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48157us (48392us total) trans 4294971923 rc 0/0 00000100:00100000:3.0:1789760709.141779:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4634 00000100:00100000:3.0:1789760709.141789:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698366075006 time 698424523667 state 0x2 ref 0 00000100:00100000:0.0:1789760709.145793:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760709.146004:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484150528 00000100:00100000:1.0:1789760709.146014:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484150528 00000100:00100000:1.0:1789760709.146217:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698366075006 time 698424523667 state 0x0 ref 1 00000100:00100000:1.0:1789760709.146234:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698424784083 time 698424523667 state 0x1 ref 1 00000100:00100000:1.0:1789760709.146288:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698424784083 time 698477218354 state 0x0 ref 1 00000100:00100000:1.0:1789760709.146301:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484150528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760709.146356:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18866176, mtime: 0x6aad94c4, atime: 0x0, ctime: 0x6aad94c4, blocks: 0x9007 00010000:00010000:1.0:1789760709.146366:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.146382:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:1.0:1789760709.146386:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760709.146392:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18866176->18870271] (req 18866176->18870271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.146405:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18866176->18870271] (req 18866176->18870271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760709.194095:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971924 is committed 00002000:00010000:1.0:1789760709.194297:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18866176 -> 18870272 00002000:00010000:1.0:1789760709.194302:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760708 -> 1789760709 00002000:00010000:1.0:1789760709.194305:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760708 -> 1789760709 00010000:00010000:1.0:1789760709.194317:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18866176->18870271] (req 18866176->18870271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.194330:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18866176->18870271] (req 18866176->18870271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.194350:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18866176->18870271] (req 18866176->18870271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.194358:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760709.194363:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18866176->18870271] (req 18866176->18870271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.194372:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18866176->18870271] (req 18866176->18870271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.194390:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18866176->18870271] (req 18866176->18870271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.194398:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962077af lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18866176->18870271] (req 18866176->18870271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760709.194590:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484150528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48286us (48801us total) trans 4294971924 rc 0/0 00000100:00100000:1.0:1789760709.194599:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4635 00000100:00100000:1.0:1789760709.194609:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698424784083 time 698477218354 state 0x2 ref 0 00000100:00100000:1.0:1789760709.199796:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760709.200023:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484150656 00000100:00100000:2.0:1789760709.200032:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484150656 00000100:00100000:2.0:1789760709.200106:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698424784083 time 698477218354 state 0x0 ref 1 00000100:00100000:2.0:1789760709.200126:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698477478770 time 698477218354 state 0x1 ref 1 00000100:00100000:2.0:1789760709.200150:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698477478770 time 698531082202 state 0x0 ref 1 00000100:00100000:2.0:1789760709.200162:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484150656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760709.200211:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18870272, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9007 00010000:00010000:2.0:1789760709.200220:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.200235:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:2.0:1789760709.200240:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760709.200246:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18870272->18874367] (req 18870272->18874367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.200260:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18870272->18874367] (req 18870272->18874367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760709.251819:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18870272 -> 18874368 00010000:00010000:1.0:1789760709.251847:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18870272->18874367] (req 18870272->18874367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.251860:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18870272->18874367] (req 18870272->18874367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.251868:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18870272->18874367] (req 18870272->18874367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.251876:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760709.251881:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18870272->18874367] (req 18870272->18874367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.251889:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18870272->18874367] (req 18870272->18874367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.251912:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18870272->18874367] (req 18870272->18874367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.252029:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962077b6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18870272->18874367] (req 18870272->18874367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760709.252218:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484150656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52052us (52426us total) trans 4294971925 rc 0/0 00000100:00100000:1.0:1789760709.252229:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4636 00000100:00100000:1.0:1789760709.252238:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698477478770 time 698531082202 state 0x2 ref 0 00000001:00080000:3.0:1789760709.255586:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971925 is committed 00000100:00100000:1.0:1789760709.259177:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760709.259321:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484150784 00000100:00100000:3.0:1789760709.259333:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484150784 00000100:00100000:3.0:1789760709.259396:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698477478770 time 698531082202 state 0x0 ref 1 00000100:00100000:3.0:1789760709.259411:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698531342618 time 698531082202 state 0x1 ref 1 00000100:00100000:3.0:1789760709.259441:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698531342618 time 698590371535 state 0x0 ref 1 00000100:00100000:3.0:1789760709.259453:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484150784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760709.259504:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18874368, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9007 00010000:00010000:3.0:1789760709.259513:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.259527:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:3.0:1789760709.259531:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760709.259538:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18874368->18878463] (req 18874368->18878463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.259552:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18874368->18878463] (req 18874368->18878463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760709.324895:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971926 is committed 00002000:00010000:3.0:1789760709.325455:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18874368 -> 18878464 00002000:00010000:3.0:1789760709.325461:0:8108:0:(ofd_lvb.c:331:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb blocks from disk: 36871 -> 38919 00010000:00010000:3.0:1789760709.325474:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18874368->18878463] (req 18874368->18878463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.325486:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18874368->18878463] (req 18874368->18878463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.325495:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18874368->18878463] (req 18874368->18878463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.325502:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760709.325507:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18874368->18878463] (req 18874368->18878463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.325516:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18874368->18878463] (req 18874368->18878463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.325536:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18874368->18878463] (req 18874368->18878463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.325543:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077bd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18874368->18878463] (req 18874368->18878463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760709.325806:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484150784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66349us (66631us total) trans 4294971926 rc 0/0 00000100:00100000:3.0:1789760709.325816:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4637 00000100:00100000:3.0:1789760709.325879:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698531342618 time 698590371535 state 0x2 ref 0 00000100:00100000:2.0:1789760709.336583:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760709.336761:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484150912 00000100:00100000:3.0:1789760709.336772:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484150912 00000100:00100000:3.0:1789760709.336825:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698531342618 time 698590371535 state 0x0 ref 1 00000100:00100000:3.0:1789760709.336852:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698590631951 time 698590371535 state 0x1 ref 1 00000100:00100000:3.0:1789760709.336882:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698590631951 time 698667812806 state 0x0 ref 1 00000100:00100000:3.0:1789760709.336894:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484150912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760709.337099:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18878464, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:3.0:1789760709.337206:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.337225:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:3.0:1789760709.337229:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760709.337235:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18878464->18882559] (req 18878464->18882559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.337250:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18878464->18882559] (req 18878464->18882559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760709.398780:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971927 is committed 00002000:00010000:1.0:1789760709.399032:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18878464 -> 18882560 00010000:00010000:1.0:1789760709.399047:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18878464->18882559] (req 18878464->18882559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.399061:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18878464->18882559] (req 18878464->18882559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.399094:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18878464->18882559] (req 18878464->18882559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.399102:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760709.399173:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18878464->18882559] (req 18878464->18882559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.399191:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18878464->18882559] (req 18878464->18882559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.399209:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18878464->18882559] (req 18878464->18882559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.399218:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962077c4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18878464->18882559] (req 18878464->18882559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760709.400625:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484150912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63713us (64031us total) trans 4294971927 rc 0/0 00000100:00100000:1.0:1789760709.400638:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4638 00000100:00100000:1.0:1789760709.400663:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698590631951 time 698667812806 state 0x2 ref 0 00000100:00100000:2.0:1789760709.418268:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760709.418451:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484151040 00000100:00100000:1.0:1789760709.418461:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484151040 00000100:00100000:1.0:1789760709.418515:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698590631951 time 698667812806 state 0x0 ref 1 00000100:00100000:1.0:1789760709.418530:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698668073222 time 698667812806 state 0x1 ref 1 00000100:00100000:1.0:1789760709.418748:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698668073222 time 698749677825 state 0x0 ref 1 00000100:00100000:1.0:1789760709.418761:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760709.418808:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18882560, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:1.0:1789760709.418817:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.418845:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:1.0:1789760709.418850:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760709.418856:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18882560->18886655] (req 18882560->18886655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.418871:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18882560->18886655] (req 18882560->18886655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760709.516313:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760709.518794:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971928 is committed 00002000:00010000:1.0:1789760709.518897:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18882560 -> 18886656 00010000:00010000:1.0:1789760709.518912:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18882560->18886655] (req 18882560->18886655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.518936:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18882560->18886655] (req 18882560->18886655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.518945:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18882560->18886655] (req 18882560->18886655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.518952:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760709.518957:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18882560->18886655] (req 18882560->18886655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.518967:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18882560->18886655] (req 18882560->18886655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.518986:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18882560->18886655] (req 18882560->18886655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.518995:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077cb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18882560->18886655] (req 18882560->18886655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760709.519122:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 100356us (100858us total) trans 4294971928 rc 0/0 00000100:00100000:1.0:1789760709.519132:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4639 00000100:00100000:1.0:1789760709.519141:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698668073222 time 698749677825 state 0x2 ref 0 00000100:00100000:2.0:1789760709.525424:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760709.525644:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484151168 00000100:00100000:1.0:1789760709.525653:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484151168 00000100:00100000:1.0:1789760709.525705:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698668073222 time 698749677825 state 0x0 ref 1 00000100:00100000:1.0:1789760709.525719:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698749938241 time 698749677825 state 0x1 ref 1 00000100:00100000:1.0:1789760709.525752:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698749938241 time 698856682658 state 0x0 ref 1 00000100:00100000:1.0:1789760709.525764:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760709.525808:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18886656, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:1.0:1789760709.525817:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.525897:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:1.0:1789760709.525903:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760709.525910:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18886656->18890751] (req 18886656->18890751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.525929:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18886656->18890751] (req 18886656->18890751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760709.587720:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971929 is committed 00002000:00010000:2.0:1789760709.587819:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18886656 -> 18890752 00010000:00010000:2.0:1789760709.587900:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18886656->18890751] (req 18886656->18890751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.587913:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18886656->18890751] (req 18886656->18890751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.587923:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18886656->18890751] (req 18886656->18890751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.587931:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760709.587935:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18886656->18890751] (req 18886656->18890751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.587944:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18886656->18890751] (req 18886656->18890751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.587964:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18886656->18890751] (req 18886656->18890751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.587972:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962077d2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18886656->18890751] (req 18886656->18890751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760709.588102:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62301us (62763us total) trans 4294971929 rc 0/0 00000100:00100000:2.0:1789760709.588112:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4640 00000100:00100000:2.0:1789760709.588122:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698749938241 time 698856682658 state 0x2 ref 0 00000100:00100000:2.0:1789760709.595230:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760709.595319:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484151296 00000100:00100000:3.0:1789760709.595331:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484151296 00000100:00100000:3.0:1789760709.595446:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698749938241 time 698856682658 state 0x0 ref 1 00000100:00100000:3.0:1789760709.595460:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698856943074 time 698856682658 state 0x1 ref 1 00000100:00100000:3.0:1789760709.595494:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698856943074 time 698926424592 state 0x0 ref 1 00000100:00100000:3.0:1789760709.595507:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760709.595564:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18890752, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:3.0:1789760709.595573:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.595590:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=698s v=(0 0) 00010000:00010000:3.0:1789760709.595594:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760709.595600:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18890752->18894847] (req 18890752->18894847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.595614:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18890752->18894847] (req 18890752->18894847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760709.709516:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971930 is committed 00002000:00010000:2.0:1789760709.709638:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18890752 -> 18894848 00010000:00010000:2.0:1789760709.709653:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18890752->18894847] (req 18890752->18894847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.709665:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18890752->18894847] (req 18890752->18894847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.709674:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18890752->18894847] (req 18890752->18894847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.709687:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760709.709692:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18890752->18894847] (req 18890752->18894847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.709701:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18890752->18894847] (req 18890752->18894847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.709722:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18890752->18894847] (req 18890752->18894847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.709730:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077d9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18890752->18894847] (req 18890752->18894847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760709.709818:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 114309us (114593us total) trans 4294971930 rc 0/0 00000100:00100000:2.0:1789760709.709827:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4641 00000100:00100000:2.0:1789760709.709836:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698856943074 time 698926424592 state 0x2 ref 0 00000100:00100000:1.0:1789760709.717543:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760709.717750:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484151424 00000100:00100000:3.0:1789760709.717762:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484151424 00000100:00100000:3.0:1789760709.717820:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698856943074 time 698926424592 state 0x0 ref 1 00000100:00100000:3.0:1789760709.717845:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698926685008 time 698926424592 state 0x1 ref 1 00000100:00100000:3.0:1789760709.717875:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698926685008 time 699048806460 state 0x0 ref 1 00000100:00100000:3.0:1789760709.717887:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7071050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484151424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760709.717941:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18894848, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:3.0:1789760709.717950:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.717965:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:3.0:1789760709.717969:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760709.717975:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18894848->18898943] (req 18894848->18898943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.717989:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18894848->18898943] (req 18894848->18898943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760709.785036:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971931 is committed 00002000:00010000:1.0:1789760709.785158:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18894848 -> 18898944 00010000:00010000:1.0:1789760709.785173:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18894848->18898943] (req 18894848->18898943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.785185:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18894848->18898943] (req 18894848->18898943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.785194:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18894848->18898943] (req 18894848->18898943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.785201:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760709.785206:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18894848->18898943] (req 18894848->18898943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.785215:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18894848->18898943] (req 18894848->18898943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.785233:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18894848->18898943] (req 18894848->18898943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760709.785240:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962077e0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18894848->18898943] (req 18894848->18898943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760709.785328:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7071050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484151424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67438us (67792us total) trans 4294971931 rc 0/0 00000100:00100000:3.0:1789760709.789238:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760709.789800:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4642 00000100:00100000:1.0:1789760709.789811:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698926685008 time 699048806460 state 0x2 ref 0 00000100:00100000:0.0:1789760709.790305:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484151552 00000100:00100000:0.0:1789760709.790314:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484151552 00000100:00100000:0.0:1789760709.790487:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 698926685008 time 699048806460 state 0x0 ref 1 00000100:00100000:0.0:1789760709.790502:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699049066876 time 699048806460 state 0x1 ref 1 00000100:00100000:0.0:1789760709.790531:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699049066876 time 699121461980 state 0x0 ref 1 00000100:00100000:0.0:1789760709.790545:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760709.790761:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18898944, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:0.0:1789760709.790771:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760709.790785:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:0.0:1789760709.790789:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760709.790796:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18898944->18903039] (req 18898944->18903039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760709.790809:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18898944->18903039] (req 18898944->18903039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760709.840187:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971932 is committed 00002000:00010000:3.0:1789760709.840294:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18898944 -> 18903040 00010000:00010000:3.0:1789760709.840309:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18898944->18903039] (req 18898944->18903039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.840321:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18898944->18903039] (req 18898944->18903039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.840331:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18898944->18903039] (req 18898944->18903039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.840516:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760709.840522:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18898944->18903039] (req 18898944->18903039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.840536:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18898944->18903039] (req 18898944->18903039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.840566:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18898944->18903039] (req 18898944->18903039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.840573:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962077e7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18898944->18903039] (req 18898944->18903039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760709.840794:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50248us (51560us total) trans 4294971932 rc 0/0 00000100:00100000:3.0:1789760709.840804:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4643 00000100:00100000:3.0:1789760709.840814:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699049066876 time 699121461980 state 0x2 ref 0 00000100:00100000:2.0:1789760709.844354:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760709.844474:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484151680 00000100:00100000:3.0:1789760709.844484:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484151680 00000100:00100000:3.0:1789760709.844532:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699049066876 time 699121461980 state 0x0 ref 1 00000100:00100000:3.0:1789760709.844546:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699121722396 time 699121461980 state 0x1 ref 1 00000100:00100000:3.0:1789760709.844657:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699121722396 time 699175513568 state 0x0 ref 1 00000100:00100000:3.0:1789760709.844673:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484151680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760709.844719:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18903040, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:3.0:1789760709.844727:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.844742:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:3.0:1789760709.844745:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760709.844752:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18903040->18907135] (req 18903040->18907135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.844765:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18903040->18907135] (req 18903040->18907135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760709.914474:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18903040 -> 18907136 00010000:00010000:3.0:1789760709.914494:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18903040->18907135] (req 18903040->18907135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.914506:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18903040->18907135] (req 18903040->18907135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.914515:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18903040->18907135] (req 18903040->18907135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.914522:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760709.914527:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18903040->18907135] (req 18903040->18907135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.914536:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18903040->18907135] (req 18903040->18907135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.914565:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18903040->18907135] (req 18903040->18907135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.914574:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962077ee lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18903040->18907135] (req 18903040->18907135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760709.914746:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484151680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 70072us (70406us total) trans 4294971933 rc 0/0 00000100:00100000:3.0:1789760709.914756:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4644 00000100:00100000:3.0:1789760709.914765:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699121722396 time 699175513568 state 0x2 ref 0 00000001:00080000:0.0:1789760709.917317:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971933 is committed 00000100:00100000:1.0:1789760709.918662:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760709.918821:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484151808 00000100:00100000:3.0:1789760709.918831:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484151808 00000100:00100000:3.0:1789760709.918884:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699121722396 time 699175513568 state 0x0 ref 1 00000100:00100000:3.0:1789760709.918897:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699175773984 time 699175513568 state 0x1 ref 1 00000100:00100000:3.0:1789760709.918928:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699175773984 time 699249858761 state 0x0 ref 1 00000100:00100000:3.0:1789760709.918939:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760709.918985:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18907136, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:3.0:1789760709.918993:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.919009:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:3.0:1789760709.919013:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760709.919019:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18907136->18911231] (req 18907136->18911231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760709.919038:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18907136->18911231] (req 18907136->18911231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760709.976223:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971934 is committed 00002000:00010000:2.0:1789760709.978846:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18907136 -> 18911232 00010000:00010000:2.0:1789760709.978866:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18907136->18911231] (req 18907136->18911231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.978878:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18907136->18911231] (req 18907136->18911231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.978889:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18907136->18911231] (req 18907136->18911231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.978897:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760709.978901:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18907136->18911231] (req 18907136->18911231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.978910:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18907136->18911231] (req 18907136->18911231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.978929:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18907136->18911231] (req 18907136->18911231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760709.978937:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962077f5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18907136->18911231] (req 18907136->18911231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760709.979034:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484151808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60092us (60450us total) trans 4294971934 rc 0/0 00000100:00100000:2.0:1789760709.979044:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4645 00000100:00100000:2.0:1789760709.979053:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699175773984 time 699249858761 state 0x2 ref 0 00000100:00100000:3.0:1789760709.990846:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760709.993869:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484151936 00000100:00100000:2.0:1789760709.993884:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484151936 00000100:00100000:0.0:1789760709.994038:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699175773984 time 699249858761 state 0x0 ref 1 00000100:00100000:0.0:1789760709.994055:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699250119177 time 699249858761 state 0x1 ref 1 00000100:00100000:0.0:1789760709.994111:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699250119177 time 699325042153 state 0x0 ref 1 00000100:00100000:0.0:1789760709.994127:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484151936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760709.994178:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18911232, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:0.0:1789760709.994188:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760709.994205:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:0.0:1789760709.994210:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760709.994217:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18911232->18915327] (req 18911232->18915327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760709.994232:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18911232->18915327] (req 18911232->18915327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760710.078482:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971935 is committed 00002000:00010000:3.0:1789760710.078731:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18911232 -> 18915328 00010000:00010000:3.0:1789760710.078747:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18911232->18915327] (req 18911232->18915327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.078760:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18911232->18915327] (req 18911232->18915327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.078768:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18911232->18915327] (req 18911232->18915327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.078776:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760710.078781:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18911232->18915327] (req 18911232->18915327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.078791:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18911232->18915327] (req 18911232->18915327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.078810:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18911232->18915327] (req 18911232->18915327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.078818:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962077fc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18911232->18915327] (req 18911232->18915327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760710.078914:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484151936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 84788us (88075us total) trans 4294971935 rc 0/0 00000100:00100000:3.0:1789760710.078924:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4646 00000100:00100000:3.0:1789760710.078933:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699250119177 time 699325042153 state 0x2 ref 0 00000100:00100000:0.0:1789760710.082543:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760710.082740:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484152064 00000100:00100000:3.0:1789760710.082752:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484152064 00000100:00100000:3.0:1789760710.082835:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699250119177 time 699325042153 state 0x0 ref 1 00000100:00100000:3.0:1789760710.082851:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699325302569 time 699325042153 state 0x1 ref 1 00000100:00100000:3.0:1789760710.082939:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699325302569 time 699413869794 state 0x0 ref 1 00000100:00100000:3.0:1789760710.082953:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484152064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760710.083002:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18915328, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:3.0:1789760710.083010:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.083026:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:3.0:1789760710.083030:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760710.083035:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18915328->18919423] (req 18915328->18919423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.083048:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18915328->18919423] (req 18915328->18919423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1789760710.083798:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:1.0:1789760710.116365:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971936 is committed 00002000:00010000:0.0:1789760710.116524:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18915328 -> 18919424 00010000:00010000:0.0:1789760710.116540:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18915328->18919423] (req 18915328->18919423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.116563:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18915328->18919423] (req 18915328->18919423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.116573:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18915328->18919423] (req 18915328->18919423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.116580:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760710.116733:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18915328->18919423] (req 18915328->18919423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.116746:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18915328->18919423] (req 18915328->18919423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.116765:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18915328->18919423] (req 18915328->18919423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.116773:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207803 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18915328->18919423] (req 18915328->18919423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760710.116902:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484152064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33947us (34364us total) trans 4294971936 rc 0/0 00000100:00100000:0.0:1789760710.116912:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4647 00000100:00100000:0.0:1789760710.116922:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699325302569 time 699413869794 state 0x2 ref 0 00000100:00100000:3.0:1789760710.122208:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760710.122290:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484152192 00000100:00100000:2.0:1789760710.122299:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484152192 00000100:00100000:2.0:1789760710.122348:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699325302569 time 699413869794 state 0x0 ref 1 00000100:00100000:2.0:1789760710.122364:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699414130210 time 699413869794 state 0x1 ref 1 00000100:00100000:2.0:1789760710.122400:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699414130210 time 699453331361 state 0x0 ref 1 00000100:00100000:2.0:1789760710.122414:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484152192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760710.122463:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18919424, mtime: 0x6aad94c5, atime: 0x0, ctime: 0x6aad94c5, blocks: 0x9807 00010000:00010000:2.0:1789760710.122472:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.122489:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:2.0:1789760710.122494:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760710.122500:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18919424->18923519] (req 18919424->18923519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.122514:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18919424->18923519] (req 18919424->18923519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760710.154838:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971937 is committed 00002000:00010000:3.0:1789760710.155150:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18919424 -> 18923520 00002000:00010000:3.0:1789760710.155155:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760709 -> 1789760710 00002000:00010000:3.0:1789760710.155159:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760709 -> 1789760710 00010000:00010000:3.0:1789760710.155172:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18919424->18923519] (req 18919424->18923519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.155185:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18919424->18923519] (req 18919424->18923519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.155195:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18919424->18923519] (req 18919424->18923519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.155202:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760710.155206:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18919424->18923519] (req 18919424->18923519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.155215:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18919424->18923519] (req 18919424->18923519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.155235:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18919424->18923519] (req 18919424->18923519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.155243:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620780a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18919424->18923519] (req 18919424->18923519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760710.155429:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484152192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33012us (33224us total) trans 4294971937 rc 0/0 00000100:00100000:3.0:1789760710.155440:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4648 00000100:00100000:3.0:1789760710.155451:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699414130210 time 699453331361 state 0x2 ref 0 00000100:00100000:0.0:1789760710.159823:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760710.159902:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484152320 00000100:00100000:1.0:1789760710.159912:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484152320 00000100:00100000:1.0:1789760710.159963:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699414130210 time 699453331361 state 0x0 ref 1 00000100:00100000:1.0:1789760710.159978:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699453591777 time 699453331361 state 0x1 ref 1 00000100:00100000:1.0:1789760710.160005:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699453591777 time 699490936249 state 0x0 ref 1 00000100:00100000:1.0:1789760710.160018:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484152320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760710.160099:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18923520, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:1.0:1789760710.160108:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.160125:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:1.0:1789760710.160130:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760710.160137:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18923520->18927615] (req 18923520->18927615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.160156:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18923520->18927615] (req 18923520->18927615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760710.191998:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971938 is committed 00002000:00010000:2.0:1789760710.192139:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18923520 -> 18927616 00010000:00010000:2.0:1789760710.192156:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18923520->18927615] (req 18923520->18927615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.192168:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18923520->18927615] (req 18923520->18927615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.192176:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18923520->18927615] (req 18923520->18927615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.192189:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760710.192193:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18923520->18927615] (req 18923520->18927615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.192201:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18923520->18927615] (req 18923520->18927615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.192219:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18923520->18927615] (req 18923520->18927615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.192226:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207811 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18923520->18927615] (req 18923520->18927615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760710.192305:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484152320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32285us (32488us total) trans 4294971938 rc 0/0 00000100:00100000:2.0:1789760710.192315:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4649 00000100:00100000:2.0:1789760710.192325:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699453591777 time 699490936249 state 0x2 ref 0 00000100:00100000:0.0:1789760710.199022:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760710.199225:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484152448 00000100:00100000:1.0:1789760710.199235:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484152448 00000100:00100000:1.0:1789760710.199296:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699453591777 time 699490936249 state 0x0 ref 1 00000100:00100000:1.0:1789760710.199312:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699491196665 time 699490936249 state 0x1 ref 1 00000100:00100000:1.0:1789760710.199573:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699491196665 time 699530502006 state 0x0 ref 1 00000100:00100000:1.0:1789760710.199676:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484152448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760710.199727:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18927616, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:1.0:1789760710.199737:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.199752:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:1.0:1789760710.199757:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760710.199763:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18927616->18931711] (req 18927616->18931711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.199779:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18927616->18931711] (req 18927616->18931711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760710.231877:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18927616 -> 18931712 00010000:00010000:0.0:1789760710.231896:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18927616->18931711] (req 18927616->18931711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.231909:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18927616->18931711] (req 18927616->18931711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.231918:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18927616->18931711] (req 18927616->18931711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.231931:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760710.231936:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18927616->18931711] (req 18927616->18931711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.231945:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18927616->18931711] (req 18927616->18931711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.231964:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18927616->18931711] (req 18927616->18931711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.231972:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207818 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18927616->18931711] (req 18927616->18931711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760710.232062:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484152448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32473us (33047us total) trans 4294971939 rc 0/0 00000100:00100000:0.0:1789760710.232102:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4650 00000100:00100000:0.0:1789760710.232112:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699491196665 time 699530502006 state 0x2 ref 0 00000001:00080000:1.0:1789760710.233173:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971939 is committed 00000100:00100000:1.0:1789760710.235408:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760710.235525:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484152576 00000100:00100000:0.0:1789760710.235535:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484152576 00000100:00100000:0.0:1789760710.235604:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699491196665 time 699530502006 state 0x0 ref 1 00000100:00100000:0.0:1789760710.235618:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699530762422 time 699530502006 state 0x1 ref 1 00000100:00100000:0.0:1789760710.235656:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699530762422 time 699566586930 state 0x0 ref 1 00000100:00100000:0.0:1789760710.235675:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484152576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760710.235723:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18931712, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:0.0:1789760710.235733:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.235758:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:0.0:1789760710.235763:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760710.235770:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18931712->18935807] (req 18931712->18935807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.235783:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18931712->18935807] (req 18931712->18935807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760710.293658:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971940 is committed 00002000:00010000:3.0:1789760710.294103:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18931712 -> 18935808 00010000:00010000:3.0:1789760710.294118:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18931712->18935807] (req 18931712->18935807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.294131:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18931712->18935807] (req 18931712->18935807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.294140:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18931712->18935807] (req 18931712->18935807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.294148:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760710.294153:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18931712->18935807] (req 18931712->18935807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.294162:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18931712->18935807] (req 18931712->18935807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.294182:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18931712->18935807] (req 18931712->18935807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.294194:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620781f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18931712->18935807] (req 18931712->18935807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760710.294278:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484152576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58603us (58874us total) trans 4294971940 rc 0/0 00000100:00100000:3.0:1789760710.294288:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4651 00000100:00100000:3.0:1789760710.294297:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699530762422 time 699566586930 state 0x2 ref 0 00000100:00100000:3.0:1789760710.298752:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760710.299098:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484152704 00000100:00100000:2.0:1789760710.299109:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484152704 00000100:00100000:2.0:1789760710.299161:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699530762422 time 699566586930 state 0x0 ref 1 00000100:00100000:2.0:1789760710.299176:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699566847346 time 699566586930 state 0x1 ref 1 00000100:00100000:2.0:1789760710.299207:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699566847346 time 699630137785 state 0x0 ref 1 00000100:00100000:2.0:1789760710.299223:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484152704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760710.299268:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18935808, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:2.0:1789760710.299276:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.299292:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:2.0:1789760710.299296:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760710.299303:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18935808->18939903] (req 18935808->18939903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.299317:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18935808->18939903] (req 18935808->18939903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760710.364455:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971941 is committed 00002000:00010000:3.0:1789760710.368794:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18935808 -> 18939904 00010000:00010000:3.0:1789760710.368815:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18935808->18939903] (req 18935808->18939903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.368827:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18935808->18939903] (req 18935808->18939903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.368836:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18935808->18939903] (req 18935808->18939903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.368856:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760710.368861:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18935808->18939903] (req 18935808->18939903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.368870:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18935808->18939903] (req 18935808->18939903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.368895:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18935808->18939903] (req 18935808->18939903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.368902:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207826 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18935808->18939903] (req 18935808->18939903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760710.369019:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484152704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69792us (70271us total) trans 4294971941 rc 0/0 00000100:00100000:3.0:1789760710.369030:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4652 00000100:00100000:3.0:1789760710.369040:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699566847346 time 699630137785 state 0x2 ref 0 00000100:00100000:0.0:1789760710.373674:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760710.373779:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484152832 00000100:00100000:1.0:1789760710.373789:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484152832 00000100:00100000:1.0:1789760710.373850:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699566847346 time 699630137785 state 0x0 ref 1 00000100:00100000:1.0:1789760710.373865:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699630398201 time 699630137785 state 0x1 ref 1 00000100:00100000:1.0:1789760710.373893:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699630398201 time 699704823928 state 0x0 ref 1 00000100:00100000:1.0:1789760710.373906:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484152832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760710.373960:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18939904, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:1.0:1789760710.373967:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.373979:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:1.0:1789760710.373982:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760710.373987:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18939904->18943999] (req 18939904->18943999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.373996:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18939904->18943999] (req 18939904->18943999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760710.445955:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18939904 -> 18944000 00010000:00010000:1.0:1789760710.445975:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18939904->18943999] (req 18939904->18943999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.445988:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18939904->18943999] (req 18939904->18943999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.445998:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18939904->18943999] (req 18939904->18943999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.446005:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760710.446010:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18939904->18943999] (req 18939904->18943999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.446019:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18939904->18943999] (req 18939904->18943999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.446038:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18939904->18943999] (req 18939904->18943999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.446046:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620782d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18939904->18943999] (req 18939904->18943999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760710.446318:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484152832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 72410us (72649us total) trans 4294971942 rc 0/0 00000100:00100000:1.0:1789760710.446341:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4653 00000100:00100000:1.0:1789760710.446353:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699630398201 time 699704823928 state 0x2 ref 0 00000001:00080000:1.0:1789760710.447965:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971942 is committed 00000100:00100000:1.0:1789760710.463372:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760710.463496:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484152960 00000100:00100000:0.0:1789760710.463506:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484152960 00000100:00100000:0.0:1789760710.463574:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699630398201 time 699704823928 state 0x0 ref 1 00000100:00100000:0.0:1789760710.463590:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699705084344 time 699704823928 state 0x1 ref 1 00000100:00100000:0.0:1789760710.463691:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699705084344 time 699794621033 state 0x0 ref 1 00000100:00100000:0.0:1789760710.463706:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484152960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760710.463754:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18944000, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:0.0:1789760710.463762:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.463778:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:0.0:1789760710.463782:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760710.463788:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18944000->18948095] (req 18944000->18948095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760710.463802:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18944000->18948095] (req 18944000->18948095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760710.530110:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971943 is committed 00002000:00010000:2.0:1789760710.530311:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18944000 -> 18948096 00010000:00010000:2.0:1789760710.530327:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18944000->18948095] (req 18944000->18948095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.530352:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18944000->18948095] (req 18944000->18948095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.530362:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18944000->18948095] (req 18944000->18948095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.530370:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760710.530375:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18944000->18948095] (req 18944000->18948095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.530386:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18944000->18948095] (req 18944000->18948095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.530407:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18944000->18948095] (req 18944000->18948095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.530415:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207834 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18944000->18948095] (req 18944000->18948095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760710.530518:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484152960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66810us (67150us total) trans 4294971943 rc 0/0 00000100:00100000:2.0:1789760710.530528:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4654 00000100:00100000:2.0:1789760710.530539:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699705084344 time 699794621033 state 0x2 ref 0 00000100:00100000:0.0:1789760710.540590:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00010000:00010000:1.0:1789760710.540938:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760710.542273:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484153088 00000100:00100000:3.0:1789760710.542286:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484153088 00000100:00100000:3.0:1789760710.542345:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699705084344 time 699794621033 state 0x0 ref 1 00000100:00100000:3.0:1789760710.542361:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699794881449 time 699794621033 state 0x1 ref 1 00000100:00100000:3.0:1789760710.542395:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699794881449 time 699873325768 state 0x0 ref 1 00000100:00100000:3.0:1789760710.542410:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484153088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760710.542458:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18948096, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:3.0:1789760710.542467:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.542482:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:3.0:1789760710.542488:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760710.542495:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18948096->18952191] (req 18948096->18952191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.542508:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18948096->18952191] (req 18948096->18952191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760710.581196:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971944 is committed 00002000:00010000:2.0:1789760710.582128:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18948096 -> 18952192 00010000:00010000:2.0:1789760710.582144:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18948096->18952191] (req 18948096->18952191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.582156:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18948096->18952191] (req 18948096->18952191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.582165:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18948096->18952191] (req 18948096->18952191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.582172:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760710.582177:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18948096->18952191] (req 18948096->18952191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.582187:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18948096->18952191] (req 18948096->18952191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.582205:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18948096->18952191] (req 18948096->18952191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.582213:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620783b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18948096->18952191] (req 18948096->18952191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760710.582303:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484153088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39893us (41719us total) trans 4294971944 rc 0/0 00000100:00100000:2.0:1789760710.582313:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4655 00000100:00100000:2.0:1789760710.582321:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699794881449 time 699873325768 state 0x2 ref 0 00000100:00100000:0.0:1789760710.585383:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760710.585448:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484153216 00000100:00100000:2.0:1789760710.585459:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484153216 00000100:00100000:2.0:1789760710.585506:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699794881449 time 699873325768 state 0x0 ref 1 00000100:00100000:2.0:1789760710.585521:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699873586184 time 699873325768 state 0x1 ref 1 00000100:00100000:2.0:1789760710.585560:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699873586184 time 699916491110 state 0x0 ref 1 00000100:00100000:2.0:1789760710.585571:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484153216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760710.585615:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18952192, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:2.0:1789760710.585623:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.585647:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:2.0:1789760710.585651:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760710.585657:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18952192->18956287] (req 18952192->18956287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.585671:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18952192->18956287] (req 18952192->18956287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760710.644532:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971945 is committed 00002000:00010000:2.0:1789760710.644650:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18952192 -> 18956288 00010000:00010000:2.0:1789760710.644664:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18952192->18956287] (req 18952192->18956287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.644677:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18952192->18956287] (req 18952192->18956287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.644686:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18952192->18956287] (req 18952192->18956287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.644693:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760710.644698:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18952192->18956287] (req 18952192->18956287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.644707:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18952192->18956287] (req 18952192->18956287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.644736:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18952192->18956287] (req 18952192->18956287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.644743:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207842 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18952192->18956287] (req 18952192->18956287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760710.644831:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484153216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59256us (59451us total) trans 4294971945 rc 0/0 00000100:00100000:2.0:1789760710.644841:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4656 00000100:00100000:2.0:1789760710.644851:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699873586184 time 699916491110 state 0x2 ref 0 00000100:00100000:1.0:1789760710.648710:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760710.648805:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484153344 00000100:00100000:2.0:1789760710.648814:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484153344 00000100:00100000:2.0:1789760710.648867:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699873586184 time 699916491110 state 0x0 ref 1 00000100:00100000:2.0:1789760710.648882:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699916751526 time 699916491110 state 0x1 ref 1 00000100:00100000:2.0:1789760710.648952:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699916751526 time 699979882660 state 0x0 ref 1 00000100:00100000:2.0:1789760710.648965:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484153344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760710.649015:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18956288, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:2.0:1789760710.649023:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.649039:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=699s v=(0 0) 00010000:00010000:2.0:1789760710.649044:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760710.649050:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18956288->18960383] (req 18956288->18960383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.649103:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18956288->18960383] (req 18956288->18960383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760710.712501:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971946 is committed 00002000:00010000:1.0:1789760710.713101:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18956288 -> 18960384 00010000:00010000:1.0:1789760710.713116:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18956288->18960383] (req 18956288->18960383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.713127:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18956288->18960383] (req 18956288->18960383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.713137:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18956288->18960383] (req 18956288->18960383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.713145:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760710.713149:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18956288->18960383] (req 18956288->18960383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.713159:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18956288->18960383] (req 18956288->18960383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.713181:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18956288->18960383] (req 18956288->18960383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.713189:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207849 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18956288->18960383] (req 18956288->18960383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760710.713318:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484153344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 64351us (64613us total) trans 4294971946 rc 0/0 00000100:00100000:1.0:1789760710.713394:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4657 00000100:00100000:1.0:1789760710.713405:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699916751526 time 699979882660 state 0x2 ref 0 00000100:00100000:0.0:1789760710.718326:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760710.719239:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484153472 00000100:00100000:1.0:1789760710.719250:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484153472 00000100:00100000:1.0:1789760710.719309:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699916751526 time 699979882660 state 0x0 ref 1 00000100:00100000:1.0:1789760710.719325:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699980143076 time 699979882660 state 0x1 ref 1 00000100:00100000:1.0:1789760710.719471:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699980143076 time 700050400677 state 0x0 ref 1 00000100:00100000:1.0:1789760710.719486:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484153472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760710.719535:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18960384, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:1.0:1789760710.719819:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.719847:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:1.0:1789760710.719853:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760710.719859:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18960384->18964479] (req 18960384->18964479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.719874:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18960384->18964479] (req 18960384->18964479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760710.776079:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971947 is committed 00002000:00010000:2.0:1789760710.776325:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18960384 -> 18964480 00010000:00010000:2.0:1789760710.776411:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18960384->18964479] (req 18960384->18964479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.776425:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18960384->18964479] (req 18960384->18964479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.776435:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18960384->18964479] (req 18960384->18964479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.776442:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760710.776447:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18960384->18964479] (req 18960384->18964479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.776456:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18960384->18964479] (req 18960384->18964479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.776474:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18960384->18964479] (req 18960384->18964479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.776482:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207850 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18960384->18964479] (req 18960384->18964479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760710.776807:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484153472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57320us (58485us total) trans 4294971947 rc 0/0 00000100:00100000:2.0:1789760710.776819:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4658 00000100:00100000:2.0:1789760710.776829:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699980143076 time 700050400677 state 0x2 ref 0 00000100:00100000:1.0:1789760710.780562:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760710.782430:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484153600 00000100:00100000:3.0:1789760710.782444:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484153600 00000100:00100000:3.0:1789760710.782506:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 699980143076 time 700050400677 state 0x0 ref 1 00000100:00100000:3.0:1789760710.782521:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700050661093 time 700050400677 state 0x1 ref 1 00000100:00100000:3.0:1789760710.782565:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700050661093 time 700113495556 state 0x0 ref 1 00000100:00100000:3.0:1789760710.782580:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484153600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760710.782966:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18964480, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:3.0:1789760710.782976:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.782992:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:3.0:1789760710.782997:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760710.783003:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18964480->18968575] (req 18964480->18968575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.783018:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18964480->18968575] (req 18964480->18968575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760710.833304:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18964480 -> 18968576 00010000:00010000:3.0:1789760710.833324:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18964480->18968575] (req 18964480->18968575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.833406:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18964480->18968575] (req 18964480->18968575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.833415:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18964480->18968575] (req 18964480->18968575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.833422:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760710.833427:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18964480->18968575] (req 18964480->18968575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.833436:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18964480->18968575] (req 18964480->18968575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.833454:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18964480->18968575] (req 18964480->18968575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.833462:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207857 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18964480->18968575] (req 18964480->18968575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760710.833564:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484153600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50983us (53017us total) trans 4294971948 rc 0/0 00000100:00100000:3.0:1789760710.833580:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4659 00000100:00100000:3.0:1789760710.833589:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700050661093 time 700113495556 state 0x2 ref 0 00000001:00080000:3.0:1789760710.837683:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971948 is committed 00000100:00100000:1.0:1789760710.837847:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760710.838910:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484153728 00000100:00100000:3.0:1789760710.838924:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484153728 00000100:00100000:3.0:1789760710.838978:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700050661093 time 700113495556 state 0x0 ref 1 00000100:00100000:3.0:1789760710.838994:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700113755972 time 700113495556 state 0x1 ref 1 00000100:00100000:3.0:1789760710.839027:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700113755972 time 700169957653 state 0x0 ref 1 00000100:00100000:3.0:1789760710.839040:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484153728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760710.839179:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18968576, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:3.0:1789760710.839189:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.839205:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:3.0:1789760710.839209:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760710.839215:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18968576->18972671] (req 18968576->18972671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.839230:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18968576->18972671] (req 18968576->18972671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760710.896499:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971949 is committed 00002000:00010000:2.0:1789760710.902564:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18968576 -> 18972672 00010000:00010000:2.0:1789760710.902585:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18968576->18972671] (req 18968576->18972671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.902599:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18968576->18972671] (req 18968576->18972671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.902610:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18968576->18972671] (req 18968576->18972671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.902618:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760710.902623:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18968576->18972671] (req 18968576->18972671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.902633:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18968576->18972671] (req 18968576->18972671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.902652:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18968576->18972671] (req 18968576->18972671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760710.902668:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620785e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18968576->18972671] (req 18968576->18972671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760710.902782:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484153728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63739us (64933us total) trans 4294971949 rc 0/0 00000100:00100000:2.0:1789760710.902793:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4660 00000100:00100000:2.0:1789760710.902802:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700113755972 time 700169957653 state 0x2 ref 0 00000100:00100000:0.0:1789760710.909413:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760710.909538:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484153856 00000100:00100000:1.0:1789760710.909559:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484153856 00000100:00100000:1.0:1789760710.909618:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700113755972 time 700169957653 state 0x0 ref 1 00000100:00100000:1.0:1789760710.909642:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700170218069 time 700169957653 state 0x1 ref 1 00000100:00100000:1.0:1789760710.909671:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700170218069 time 700240602046 state 0x0 ref 1 00000100:00100000:1.0:1789760710.909692:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484153856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760710.909743:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18972672, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:1.0:1789760710.909751:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.909766:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:1.0:1789760710.909770:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760710.909776:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18972672->18976767] (req 18972672->18976767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.909790:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18972672->18976767] (req 18972672->18976767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760710.977503:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971950 is committed 00002000:00010000:3.0:1789760710.977847:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18972672 -> 18976768 00010000:00010000:3.0:1789760710.977863:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18972672->18976767] (req 18972672->18976767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.977876:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18972672->18976767] (req 18972672->18976767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.977885:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18972672->18976767] (req 18972672->18976767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.977892:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760710.977897:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18972672->18976767] (req 18972672->18976767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.977907:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18972672->18976767] (req 18972672->18976767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.977926:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18972672->18976767] (req 18972672->18976767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760710.977934:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207865 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18972672->18976767] (req 18972672->18976767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760710.978086:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484153856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68373us (68657us total) trans 4294971950 rc 0/0 00000100:00100000:3.0:1789760710.978096:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4661 00000100:00100000:3.0:1789760710.978107:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700170218069 time 700240602046 state 0x2 ref 0 00000100:00100000:0.0:1789760710.986485:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760710.986813:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484153984 00000100:00100000:1.0:1789760710.986824:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484153984 00000100:00100000:1.0:1789760710.986879:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700170218069 time 700240602046 state 0x0 ref 1 00000100:00100000:1.0:1789760710.986895:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700240862462 time 700240602046 state 0x1 ref 1 00000100:00100000:1.0:1789760710.986963:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700240862462 time 700317893152 state 0x0 ref 1 00000100:00100000:1.0:1789760710.986977:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484153984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760710.987027:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18976768, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:1.0:1789760710.987036:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.987053:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:1.0:1789760710.987057:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760710.987094:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18976768->18980863] (req 18976768->18980863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760710.987111:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18976768->18980863] (req 18976768->18980863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760711.053161:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f45c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508906752:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.053230:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.053277:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501ba0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508907264:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.053302:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.053318:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb724e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508907776:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.053350:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.053372:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:3.0:1789760711.053378:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f47b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508906368:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.053404:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.053426:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f47480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508906880:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.053448:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.053466:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501ba1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508907392:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.053486:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.053501:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb725880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508907904:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.053524:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.053562:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f46d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508906496:0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.053785:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f46680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508906624:0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.053816:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760711.053836:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f44000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0000_UUID:3481:1876699508907136:0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.053865:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508907264 00000100:00100000:2.0:1789760711.053871:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508907264 00000100:00100000:2.0:1789760711.053921:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508907776 00000100:00100000:2.0:1789760711.053924:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508907776 00000100:00100000:2.0:1789760711.053938:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508907392 00000100:00100000:2.0:1789760711.053941:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508907392 00000100:00100000:2.0:1789760711.053951:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508907904 00000100:00100000:2.0:1789760711.053953:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508907904 00000100:00100000:2.0:1789760711.053967:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 302 00000100:00100000:2.0:1789760711.053976:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508907264:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054034:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508907264:12345-0@lo:400:kworker.0 Request processed in 59us (731us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.054041:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 302 00000100:00100000:2.0:1789760711.054059:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 303 00000100:00100000:2.0:1789760711.054082:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508907776:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054111:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508907776:12345-0@lo:400:kworker.0 Request processed in 46us (760us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.054118:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 303 00000100:00100000:2.0:1789760711.054131:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 304 00000100:00100000:2.0:1789760711.054136:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508907392:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054161:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508907392:12345-0@lo:400:kworker.0 Request processed in 25us (674us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.054167:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 304 00000100:00100000:2.0:1789760711.054177:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 305 00000100:00100000:2.0:1789760711.054183:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508907904:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054208:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508907904:12345-0@lo:400:kworker.0 Request processed in 25us (682us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.054214:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 305 00000100:00100000:2.0:1789760711.054279:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508906880 00000100:00100000:2.0:1789760711.054283:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508906880 00000100:00100000:2.0:1789760711.054305:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508906624 00000100:00100000:2.0:1789760711.054308:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508906624 00000100:00100000:2.0:1789760711.054320:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 415 00000100:00100000:2.0:1789760711.054326:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb726a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508906880:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054360:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb726a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508906880:12345-0@lo:400:kworker.0 Request processed in 34us (911us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.054366:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 415 00000100:00100000:2.0:1789760711.054377:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 416 00000100:00100000:2.0:1789760711.054382:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8380 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3481:x1876699508906624:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054408:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8380 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3481:x1876699508906624:12345-0@lo:400:kworker.0 Request processed in 25us (590us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.054414:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 416 00000100:00080000:2.0:1789760711.054429:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:2.0:1789760711.054433:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760711.054468:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f47480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508906880:0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054480:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501ba1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508907392:0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054489:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb725880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508907904:0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054497:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501ba0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508907264:0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.054507:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb724e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508907776:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.054904:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.054939:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f45f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508907008:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.054968:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.054988:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501ba0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508907520:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.055009:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.055028:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508908032:0@lo:13:osp-pre-1-1.0 00000100:00100000:3.0:1789760711.055049:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.055081:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1789760711.055087:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508908160:0@lo:13:osp-pre-1-0.0 00000100:00100000:3.0:1789760711.055108:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760711.055136:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508906752 00000100:00100000:3.0:1789760711.055142:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508906752 00000100:00100000:3.0:1789760711.055168:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508906496 00000100:00100000:3.0:1789760711.055171:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508906496 00000100:00100000:3.0:1789760711.055184:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508907008 00000100:00100000:3.0:1789760711.055187:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508907008 00000100:00100000:3.0:1789760711.055202:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508907520 00000100:00100000:3.0:1789760711.055205:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508907520 00000100:00100000:3.0:1789760711.055220:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 855 00000100:00100000:3.0:1789760711.055226:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb727100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508906752:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.055264:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb727100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508906752:12345-0@lo:400:kworker.0 Request processed in 38us (2034us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760711.055270:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 855 00000100:00100000:3.0:1789760711.055285:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 856 00000100:00100000:3.0:1789760711.055305:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886dc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508906496:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.055347:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886dc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508906496:12345-0@lo:400:kworker.0 Request processed in 56us (443us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760711.055352:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 856 00000100:00100000:3.0:1789760711.055364:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 857 00000100:00100000:3.0:1789760711.055370:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508907008:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.055392:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508907008:12345-0@lo:400:kworker.0 Request processed in 24us (425us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760711.055398:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 857 00000100:00100000:3.0:1789760711.055408:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 858 00000100:00100000:3.0:1789760711.055413:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508907520:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.055435:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508907520:12345-0@lo:400:kworker.0 Request processed in 22us (426us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760711.055440:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 858 00000100:00100000:3.0:1789760711.055459:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508906368 00000100:00100000:3.0:1789760711.055463:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508906368 00000100:00100000:3.0:1789760711.055486:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 466 00000100:00100000:3.0:1789760711.055493:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4d2911050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508906368:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.055522:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4d2911050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508906368:12345-0@lo:400:kworker.0 Request processed in 30us (2117us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760711.055528:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 466 00000100:00100000:3.0:1789760711.055578:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508908032 00000100:00100000:3.0:1789760711.055590:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508908032 00000100:00100000:2.0:1789760711.056102:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508908160 00000100:00100000:2.0:1789760711.056107:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508908160 00000100:00100000:2.0:1789760711.056139:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 356 00000100:00100000:2.0:1789760711.056146:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508908160:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1789760711.056176:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760711.056218:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508908160:12345-0@lo:13:osp-pre-1-0.0 Request processed in 71us (1108us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.056224:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 356 00000100:00080000:2.0:1789760711.056249:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:2.0:1789760711.056277:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f47b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508906368:0@lo:400:kworker.0 00000100:00100000:2.0:1789760711.056292:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f45c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508906752:0@lo:400:kworker.0 00000100:00100000:1.0:1789760711.056387:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760711.056421:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb727b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0001_UUID:3481:1876699508907648:0@lo:400:kworker.0 00000100:00100000:1.0:1789760711.056450:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760711.056481:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f46680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508906624:0@lo:400:kworker.0 00000100:00100000:1.0:1789760711.056501:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508907136 00000100:00100000:1.0:1789760711.056505:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508907136 00000100:00100000:1.0:1789760711.056530:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508907648 00000100:00100000:1.0:1789760711.056533:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508907648 00000100:00100000:1.0:1789760711.056568:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 859 00000100:00100000:1.0:1789760711.056575:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0000_UUID+5:3481:x1876699508907136:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760711.056631:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0000_UUID+5:3481:x1876699508907136:12345-0@lo:400:kworker.0 Request processed in 56us (243us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760711.056640:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 859 00000100:00100000:1.0:1789760711.056652:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 860 00000100:00100000:1.0:1789760711.056658:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8911f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0001_UUID+5:3481:x1876699508907648:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760711.057109:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8911f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0001_UUID+5:3481:x1876699508907648:12345-0@lo:400:kworker.0 Request processed in 451us (658us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760711.057116:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 860 00000100:00100000:1.0:1789760711.057147:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f44000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0000_UUID:3481:1876699508907136:0@lo:400:kworker.0 00000100:00100000:1.0:1789760711.057159:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb727b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0001_UUID:3481:1876699508907648:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.057505:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 357 00000100:00100000:3.0:1789760711.057515:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508908032:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:3.0:1789760711.057531:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:3.0:1789760711.057588:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508908032:12345-0@lo:13:osp-pre-1-1.0 Request processed in 66us (2528us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760711.057595:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 357 00000100:00100000:3.0:1789760711.057642:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508908160:0@lo:13:osp-pre-1-0.0 00000100:00100000:3.0:1789760711.057655:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508908032:0@lo:13:osp-pre-1-1.0 00000100:00100000:3.0:1789760711.057665:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f46d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508906496:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.057685:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f45f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508907008:0@lo:400:kworker.0 00000100:00100000:3.0:1789760711.057697:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501ba0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508907520:0@lo:400:kworker.0 00000100:00080000:2.0:1789760711.057782:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000001:00080000:2.0:1789760711.061278:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971951 is committed 00002000:00010000:0.0:1789760711.063298:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18976768 -> 18980864 00010000:00010000:0.0:1789760711.063318:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18976768->18980863] (req 18976768->18980863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.063331:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18976768->18980863] (req 18976768->18980863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.063339:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18976768->18980863] (req 18976768->18980863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.063348:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760711.063352:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18976768->18980863] (req 18976768->18980863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.063362:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18976768->18980863] (req 18976768->18980863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.063381:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18976768->18980863] (req 18976768->18980863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.063388:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620786c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18976768->18980863] (req 18976768->18980863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760711.064136:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484153984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77155us (77655us total) trans 4294971951 rc 0/0 00000100:00100000:0.0:1789760711.064147:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4662 00000100:00100000:0.0:1789760711.064157:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700240862462 time 700317893152 state 0x2 ref 0 00000100:00100000:1.0:1789760711.069532:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760711.069862:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484154112 00000100:00100000:3.0:1789760711.069873:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484154112 00000100:00100000:3.0:1789760711.069924:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700240862462 time 700317893152 state 0x0 ref 1 00000100:00100000:3.0:1789760711.069938:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700318153568 time 700317893152 state 0x1 ref 1 00000100:00100000:3.0:1789760711.069972:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700318153568 time 700400902779 state 0x0 ref 1 00000100:00100000:3.0:1789760711.069985:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484154112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760711.070031:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18980864, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:3.0:1789760711.070040:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.070055:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:3.0:1789760711.070060:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760711.070302:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18980864->18984959] (req 18980864->18984959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.070319:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18980864->18984959] (req 18980864->18984959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760711.117503:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971952 is committed 00002000:00010000:3.0:1789760711.117614:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18980864 -> 18984960 00010000:00010000:3.0:1789760711.117638:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18980864->18984959] (req 18980864->18984959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.117652:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18980864->18984959] (req 18980864->18984959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.117666:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18980864->18984959] (req 18980864->18984959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.117674:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760711.117679:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18980864->18984959] (req 18980864->18984959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.117688:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18980864->18984959] (req 18980864->18984959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.117707:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18980864->18984959] (req 18980864->18984959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.117715:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207873 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18980864->18984959] (req 18980864->18984959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760711.117806:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484154112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47820us (48277us total) trans 4294971952 rc 0/0 00000100:00100000:3.0:1789760711.117816:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4663 00000100:00100000:3.0:1789760711.117825:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700318153568 time 700400902779 state 0x2 ref 0 00000100:00100000:3.0:1789760711.123644:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760711.123768:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484154240 00000100:00100000:2.0:1789760711.123779:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484154240 00000100:00100000:2.0:1789760711.123850:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700318153568 time 700400902779 state 0x0 ref 1 00000100:00100000:2.0:1789760711.123866:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700401163195 time 700400902779 state 0x1 ref 1 00000100:00100000:2.0:1789760711.123898:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700401163195 time 700454829193 state 0x0 ref 1 00000100:00100000:2.0:1789760711.123910:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484154240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760711.123957:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18984960, mtime: 0x6aad94c6, atime: 0x0, ctime: 0x6aad94c6, blocks: 0x9807 00010000:00010000:2.0:1789760711.123966:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.123982:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:2.0:1789760711.123986:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760711.123992:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18984960->18989055] (req 18984960->18989055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.124006:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18984960->18989055] (req 18984960->18989055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760711.186792:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971953 is committed 00002000:00010000:2.0:1789760711.189855:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18984960 -> 18989056 00002000:00010000:2.0:1789760711.189864:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760710 -> 1789760711 00002000:00010000:2.0:1789760711.189867:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760710 -> 1789760711 00010000:00010000:2.0:1789760711.189881:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18984960->18989055] (req 18984960->18989055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.189895:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18984960->18989055] (req 18984960->18989055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.189904:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18984960->18989055] (req 18984960->18989055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.189912:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760711.189918:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18984960->18989055] (req 18984960->18989055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.189928:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18984960->18989055] (req 18984960->18989055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.189947:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18984960->18989055] (req 18984960->18989055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.189956:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620787a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18984960->18989055] (req 18984960->18989055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760711.190243:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484154240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66329us (66610us total) trans 4294971953 rc 0/0 00000100:00100000:2.0:1789760711.190253:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4664 00000100:00100000:2.0:1789760711.190263:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700401163195 time 700454829193 state 0x2 ref 0 00000100:00100000:2.0:1789760711.196785:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760711.196874:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484154368 00000100:00100000:1.0:1789760711.196886:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484154368 00000100:00100000:1.0:1789760711.197302:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700401163195 time 700454829193 state 0x0 ref 1 00000100:00100000:1.0:1789760711.197318:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700455089609 time 700454829193 state 0x1 ref 1 00000100:00100000:1.0:1789760711.197452:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700455089609 time 700528381941 state 0x0 ref 1 00000100:00100000:1.0:1789760711.197467:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484154368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760711.197516:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18989056, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:1.0:1789760711.197524:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.197538:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:1.0:1789760711.197553:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760711.197561:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18989056->18993151] (req 18989056->18993151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.197574:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18989056->18993151] (req 18989056->18993151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760711.249954:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971954 is committed 00002000:00010000:1.0:1789760711.249996:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18989056 -> 18993152 00010000:00010000:1.0:1789760711.250011:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18989056->18993151] (req 18989056->18993151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.250028:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18989056->18993151] (req 18989056->18993151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.250037:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18989056->18993151] (req 18989056->18993151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.250045:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760711.250050:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18989056->18993151] (req 18989056->18993151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.250059:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18989056->18993151] (req 18989056->18993151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.250233:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18989056->18993151] (req 18989056->18993151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.250241:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207881 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18989056->18993151] (req 18989056->18993151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760711.250496:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484154368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53027us (53714us total) trans 4294971954 rc 0/0 00000100:00100000:0.0:1789760711.250506:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4665 00000100:00100000:0.0:1789760711.250515:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700455089609 time 700528381941 state 0x2 ref 0 00000100:00100000:3.0:1789760711.255203:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760711.255273:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484154496 00000100:00100000:0.0:1789760711.255284:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484154496 00000100:00100000:0.0:1789760711.255359:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700455089609 time 700528381941 state 0x0 ref 1 00000100:00100000:0.0:1789760711.255377:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700528642357 time 700528381941 state 0x1 ref 1 00000100:00100000:0.0:1789760711.255410:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700528642357 time 700586340706 state 0x0 ref 1 00000100:00100000:0.0:1789760711.255426:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484154496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760711.255478:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18993152, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:0.0:1789760711.255487:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.255503:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:0.0:1789760711.255507:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760711.255514:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18993152->18997247] (req 18993152->18997247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.255529:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18993152->18997247] (req 18993152->18997247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760711.292422:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971955 is committed 00002000:00010000:1.0:1789760711.292526:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18993152 -> 18997248 00010000:00010000:1.0:1789760711.292551:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18993152->18997247] (req 18993152->18997247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.292563:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18993152->18997247] (req 18993152->18997247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.292572:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18993152->18997247] (req 18993152->18997247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.292578:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760711.292582:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18993152->18997247] (req 18993152->18997247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.292591:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18993152->18997247] (req 18993152->18997247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.292617:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18993152->18997247] (req 18993152->18997247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.292625:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207888 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18993152->18997247] (req 18993152->18997247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760711.292823:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484154496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37398us (37625us total) trans 4294971955 rc 0/0 00000100:00100000:1.0:1789760711.292837:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4666 00000100:00100000:1.0:1789760711.292847:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700528642357 time 700586340706 state 0x2 ref 0 00000100:00100000:1.0:1789760711.297256:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760711.297363:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484154624 00000100:00100000:2.0:1789760711.297373:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484154624 00000100:00100000:2.0:1789760711.297424:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700528642357 time 700586340706 state 0x0 ref 1 00000100:00100000:2.0:1789760711.297437:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700586601122 time 700586340706 state 0x1 ref 1 00000100:00100000:2.0:1789760711.297467:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700586601122 time 700628398918 state 0x0 ref 1 00000100:00100000:2.0:1789760711.297479:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484154624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760711.297525:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 18997248, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:2.0:1789760711.297533:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.297558:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:2.0:1789760711.297562:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760711.297569:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18997248->19001343] (req 18997248->19001343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.297582:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18997248->19001343] (req 18997248->19001343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760711.341408:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971956 is committed 00002000:00010000:1.0:1789760711.341530:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 18997248 -> 19001344 00010000:00010000:1.0:1789760711.341559:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18997248->19001343] (req 18997248->19001343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.341572:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18997248->19001343] (req 18997248->19001343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.341582:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18997248->19001343] (req 18997248->19001343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.341590:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760711.341594:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18997248->19001343] (req 18997248->19001343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.341612:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18997248->19001343] (req 18997248->19001343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.341638:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18997248->19001343] (req 18997248->19001343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.341647:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620788f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [18997248->19001343] (req 18997248->19001343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760711.341743:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484154624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44261us (44492us total) trans 4294971956 rc 0/0 00000100:00100000:1.0:1789760711.341754:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4667 00000100:00100000:1.0:1789760711.341763:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700586601122 time 700628398918 state 0x2 ref 0 00000100:00100000:2.0:1789760711.347803:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760711.347924:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484154752 00000100:00100000:3.0:1789760711.347934:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484154752 00000100:00100000:3.0:1789760711.347993:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700586601122 time 700628398918 state 0x0 ref 1 00000100:00100000:3.0:1789760711.348007:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700628659334 time 700628398918 state 0x1 ref 1 00000100:00100000:3.0:1789760711.348034:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700628659334 time 700678965265 state 0x0 ref 1 00000100:00100000:3.0:1789760711.348045:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484154752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760711.348131:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19001344, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:3.0:1789760711.348139:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.348154:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:3.0:1789760711.348158:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760711.348164:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19001344->19005439] (req 19001344->19005439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.348178:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19001344->19005439] (req 19001344->19005439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760711.395289:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19001344 -> 19005440 00010000:00010000:1.0:1789760711.395308:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19001344->19005439] (req 19001344->19005439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.395321:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19001344->19005439] (req 19001344->19005439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.395406:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19001344->19005439] (req 19001344->19005439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.395414:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760711.395419:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19001344->19005439] (req 19001344->19005439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.395430:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19001344->19005439] (req 19001344->19005439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.395451:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19001344->19005439] (req 19001344->19005439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.395465:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207896 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19001344->19005439] (req 19001344->19005439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760711.395571:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484154752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47523us (47772us total) trans 4294971957 rc 0/0 00000100:00100000:1.0:1789760711.395583:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4668 00000100:00100000:1.0:1789760711.395593:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700628659334 time 700678965265 state 0x2 ref 0 00000001:00080000:2.0:1789760711.395859:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971957 is committed 00000100:00100000:0.0:1789760711.400318:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760711.400463:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484154880 00000100:00100000:3.0:1789760711.400473:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484154880 00000100:00100000:3.0:1789760711.400528:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700628659334 time 700678965265 state 0x0 ref 1 00000100:00100000:3.0:1789760711.400557:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700679225681 time 700678965265 state 0x1 ref 1 00000100:00100000:3.0:1789760711.400637:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700679225681 time 700731558087 state 0x0 ref 1 00000100:00100000:3.0:1789760711.400654:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484154880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760711.400700:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19005440, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:3.0:1789760711.400710:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.400727:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:3.0:1789760711.400732:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760711.400738:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19005440->19009535] (req 19005440->19009535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.400753:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19005440->19009535] (req 19005440->19009535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760711.446306:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971958 is committed 00002000:00010000:0.0:1789760711.446466:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19005440 -> 19009536 00010000:00010000:0.0:1789760711.446481:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19005440->19009535] (req 19005440->19009535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.446493:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19005440->19009535] (req 19005440->19009535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.446502:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19005440->19009535] (req 19005440->19009535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.446509:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760711.446514:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19005440->19009535] (req 19005440->19009535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.446522:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19005440->19009535] (req 19005440->19009535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.446548:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19005440->19009535] (req 19005440->19009535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.446557:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620789d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19005440->19009535] (req 19005440->19009535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760711.446711:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484154880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46055us (46397us total) trans 4294971958 rc 0/0 00000100:00100000:0.0:1789760711.446721:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4669 00000100:00100000:0.0:1789760711.446730:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700679225681 time 700731558087 state 0x2 ref 0 00000100:00100000:3.0:1789760711.450966:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760711.451827:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484155008 00000100:00100000:0.0:1789760711.451839:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484155008 00000100:00100000:0.0:1789760711.451916:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700679225681 time 700731558087 state 0x0 ref 1 00000100:00100000:0.0:1789760711.451932:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700731818503 time 700731558087 state 0x1 ref 1 00000100:00100000:0.0:1789760711.452108:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700731818503 time 700783037500 state 0x0 ref 1 00000100:00100000:0.0:1789760711.452123:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484155008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760711.452173:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19009536, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:0.0:1789760711.452182:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.452197:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:0.0:1789760711.452202:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760711.452208:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19009536->19013631] (req 19009536->19013631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.452221:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19009536->19013631] (req 19009536->19013631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760711.486715:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971959 is committed 00002000:00010000:2.0:1789760711.486852:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19009536 -> 19013632 00010000:00010000:2.0:1789760711.486867:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19009536->19013631] (req 19009536->19013631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.486880:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19009536->19013631] (req 19009536->19013631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.486890:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19009536->19013631] (req 19009536->19013631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.486898:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760711.486903:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19009536->19013631] (req 19009536->19013631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.486913:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19009536->19013631] (req 19009536->19013631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.486931:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19009536->19013631] (req 19009536->19013631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.486940:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962078a4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19009536->19013631] (req 19009536->19013631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760711.487164:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484155008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35040us (36203us total) trans 4294971959 rc 0/0 00000100:00100000:2.0:1789760711.487175:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4670 00000100:00100000:2.0:1789760711.487185:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700731818503 time 700783037500 state 0x2 ref 0 00000100:00100000:1.0:1789760711.491449:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760711.491826:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484155136 00000100:00100000:2.0:1789760711.491836:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484155136 00000100:00100000:2.0:1789760711.491888:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700731818503 time 700783037500 state 0x0 ref 1 00000100:00100000:2.0:1789760711.491902:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700783297916 time 700783037500 state 0x1 ref 1 00000100:00100000:2.0:1789760711.491934:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700783297916 time 700822864392 state 0x0 ref 1 00000100:00100000:2.0:1789760711.491946:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484155136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760711.492213:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19013632, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:2.0:1789760711.492223:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.492239:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:2.0:1789760711.492244:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760711.492251:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19013632->19017727] (req 19013632->19017727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.492266:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19013632->19017727] (req 19013632->19017727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760711.526831:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19013632 -> 19017728 00010000:00010000:2.0:1789760711.526850:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19013632->19017727] (req 19013632->19017727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.526862:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19013632->19017727] (req 19013632->19017727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.526871:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19013632->19017727] (req 19013632->19017727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.526878:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760711.526884:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19013632->19017727] (req 19013632->19017727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.526893:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19013632->19017727] (req 19013632->19017727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.526912:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19013632->19017727] (req 19013632->19017727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.526919:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962078ab lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19013632->19017727] (req 19013632->19017727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760711.527019:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484155136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35069us (35575us total) trans 4294971960 rc 0/0 00000100:00100000:2.0:1789760711.527030:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4671 00000100:00100000:2.0:1789760711.527039:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700783297916 time 700822864392 state 0x2 ref 0 00000001:00080000:3.0:1789760711.527255:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971960 is committed 00000100:00100000:3.0:1789760711.530691:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760711.531236:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484155264 00000100:00100000:1.0:1789760711.531247:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484155264 00000100:00100000:1.0:1789760711.531301:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700783297916 time 700822864392 state 0x0 ref 1 00000100:00100000:1.0:1789760711.531316:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700823124808 time 700822864392 state 0x1 ref 1 00000100:00100000:1.0:1789760711.531386:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700823124808 time 700862315581 state 0x0 ref 1 00000100:00100000:1.0:1789760711.531398:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484155264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760711.531444:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19017728, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:1.0:1789760711.531453:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.531471:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:1.0:1789760711.531475:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760711.531481:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19017728->19021823] (req 19017728->19021823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.531495:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19017728->19021823] (req 19017728->19021823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.570711:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760711.574947:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971961 is committed 00002000:00010000:1.0:1789760711.575199:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19017728 -> 19021824 00010000:00010000:1.0:1789760711.575219:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19017728->19021823] (req 19017728->19021823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.575232:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19017728->19021823] (req 19017728->19021823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.575241:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19017728->19021823] (req 19017728->19021823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.575249:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760711.575254:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19017728->19021823] (req 19017728->19021823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.575263:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19017728->19021823] (req 19017728->19021823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.575283:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19017728->19021823] (req 19017728->19021823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.575291:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078b2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19017728->19021823] (req 19017728->19021823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760711.575425:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484155264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44025us (44739us total) trans 4294971961 rc 0/0 00000100:00100000:1.0:1789760711.575436:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4672 00000100:00100000:1.0:1789760711.575446:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700823124808 time 700862315581 state 0x2 ref 0 00000100:00100000:1.0:1789760711.579844:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760711.580020:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484155392 00000100:00100000:3.0:1789760711.580029:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484155392 00000100:00100000:3.0:1789760711.580227:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700823124808 time 700862315581 state 0x0 ref 1 00000100:00100000:3.0:1789760711.580242:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700862575997 time 700862315581 state 0x1 ref 1 00000100:00100000:3.0:1789760711.580275:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700862575997 time 700911206519 state 0x0 ref 1 00000100:00100000:3.0:1789760711.580288:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484155392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760711.580346:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19021824, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:3.0:1789760711.580356:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.580372:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:3.0:1789760711.580377:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760711.580383:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19021824->19025919] (req 19021824->19025919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.580397:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19021824->19025919] (req 19021824->19025919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760711.616308:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971962 is committed 00002000:00010000:0.0:1789760711.616485:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19021824 -> 19025920 00010000:00010000:0.0:1789760711.616499:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19021824->19025919] (req 19021824->19025919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.616512:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19021824->19025919] (req 19021824->19025919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.616522:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19021824->19025919] (req 19021824->19025919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.616535:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760711.616551:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19021824->19025919] (req 19021824->19025919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.616561:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19021824->19025919] (req 19021824->19025919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.616578:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19021824->19025919] (req 19021824->19025919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.616585:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078b9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19021824->19025919] (req 19021824->19025919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760711.616862:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484155392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36571us (37022us total) trans 4294971962 rc 0/0 00000100:00100000:0.0:1789760711.616872:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4673 00000100:00100000:0.0:1789760711.616882:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700862575997 time 700911206519 state 0x2 ref 0 00000100:00100000:0.0:1789760711.622049:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760711.622139:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484155520 00000100:00100000:1.0:1789760711.622149:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484155520 00000100:00100000:1.0:1789760711.622206:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700862575997 time 700911206519 state 0x0 ref 1 00000100:00100000:1.0:1789760711.622221:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700911466935 time 700911206519 state 0x1 ref 1 00000100:00100000:1.0:1789760711.622251:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700911466935 time 700953181879 state 0x0 ref 1 00000100:00100000:1.0:1789760711.622265:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484155520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760711.622313:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19025920, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:1.0:1789760711.622331:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.622347:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:1.0:1789760711.622352:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760711.622358:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19025920->19030015] (req 19025920->19030015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.622378:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19025920->19030015] (req 19025920->19030015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760711.660534:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971963 is committed 00002000:00010000:2.0:1789760711.660841:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19025920 -> 19030016 00010000:00010000:2.0:1789760711.660857:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19025920->19030015] (req 19025920->19030015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.660870:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19025920->19030015] (req 19025920->19030015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.660880:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19025920->19030015] (req 19025920->19030015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.660894:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760711.660899:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19025920->19030015] (req 19025920->19030015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.660909:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19025920->19030015] (req 19025920->19030015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.660929:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19025920->19030015] (req 19025920->19030015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.660937:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962078c0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19025920->19030015] (req 19025920->19030015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760711.661168:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484155520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38901us (39122us total) trans 4294971963 rc 0/0 00000100:00100000:2.0:1789760711.661178:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4674 00000100:00100000:2.0:1789760711.661187:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700911466935 time 700953181879 state 0x2 ref 0 00000100:00100000:0.0:1789760711.664880:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760711.665593:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484155648 00000100:00100000:2.0:1789760711.665604:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484155648 00000100:00100000:2.0:1789760711.665713:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700911466935 time 700953181879 state 0x0 ref 1 00000100:00100000:2.0:1789760711.665728:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700953442295 time 700953181879 state 0x1 ref 1 00000100:00100000:2.0:1789760711.665854:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700953442295 time 700996780764 state 0x0 ref 1 00000100:00100000:2.0:1789760711.665871:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484155648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760711.665925:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19030016, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:2.0:1789760711.665934:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.665951:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=700s v=(0 0) 00010000:00010000:2.0:1789760711.665956:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760711.665962:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19030016->19034111] (req 19030016->19034111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.666046:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19030016->19034111] (req 19030016->19034111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760711.703569:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971964 is committed 00002000:00010000:2.0:1789760711.703725:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19030016 -> 19034112 00010000:00010000:2.0:1789760711.703740:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19030016->19034111] (req 19030016->19034111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.703753:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19030016->19034111] (req 19030016->19034111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.703762:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19030016->19034111] (req 19030016->19034111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.703778:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760711.703783:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19030016->19034111] (req 19030016->19034111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.703793:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19030016->19034111] (req 19030016->19034111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.703815:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19030016->19034111] (req 19030016->19034111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.703823:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078c7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19030016->19034111] (req 19030016->19034111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760711.703906:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484155648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38034us (39031us total) trans 4294971964 rc 0/0 00000100:00100000:2.0:1789760711.703916:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4675 00000100:00100000:2.0:1789760711.703926:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700953442295 time 700996780764 state 0x2 ref 0 00000100:00100000:1.0:1789760711.708288:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760711.708474:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484155776 00000100:00100000:2.0:1789760711.708484:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484155776 00000100:00100000:2.0:1789760711.708534:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700953442295 time 700996780764 state 0x0 ref 1 00000100:00100000:2.0:1789760711.708559:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700997041180 time 700996780764 state 0x1 ref 1 00000100:00100000:2.0:1789760711.708614:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700997041180 time 701039544722 state 0x0 ref 1 00000100:00100000:2.0:1789760711.708637:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484155776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760711.708685:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19034112, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:2.0:1789760711.708694:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.708711:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:2.0:1789760711.708715:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760711.708720:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19034112->19038207] (req 19034112->19038207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.708734:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19034112->19038207] (req 19034112->19038207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760711.741458:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971965 is committed 00002000:00010000:2.0:1789760711.741566:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19034112 -> 19038208 00010000:00010000:2.0:1789760711.741581:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19034112->19038207] (req 19034112->19038207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.741594:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19034112->19038207] (req 19034112->19038207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.741603:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19034112->19038207] (req 19034112->19038207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.741610:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760711.741615:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19034112->19038207] (req 19034112->19038207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.741692:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19034112->19038207] (req 19034112->19038207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.741715:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19034112->19038207] (req 19034112->19038207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.741722:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078ce lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19034112->19038207] (req 19034112->19038207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760711.741922:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484155776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33292us (33637us total) trans 4294971965 rc 0/0 00000100:00100000:2.0:1789760711.741932:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4676 00000100:00100000:2.0:1789760711.741941:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700997041180 time 701039544722 state 0x2 ref 0 00000100:00100000:0.0:1789760711.746908:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760711.747037:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484155904 00000100:00100000:3.0:1789760711.747046:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484155904 00000100:00100000:3.0:1789760711.748913:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 700997041180 time 701039544722 state 0x0 ref 1 00000100:00100000:3.0:1789760711.748931:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701039805138 time 701039544722 state 0x1 ref 1 00000100:00100000:3.0:1789760711.749138:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701039805138 time 701080067659 state 0x0 ref 1 00000100:00100000:3.0:1789760711.749155:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484155904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760711.749205:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19038208, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:3.0:1789760711.749213:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.749232:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:3.0:1789760711.749236:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760711.749242:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19038208->19042303] (req 19038208->19042303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.749255:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19038208->19042303] (req 19038208->19042303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760711.787005:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19038208 -> 19042304 00010000:00010000:1.0:1789760711.787024:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19038208->19042303] (req 19038208->19042303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.787038:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19038208->19042303] (req 19038208->19042303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.787047:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19038208->19042303] (req 19038208->19042303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.787055:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760711.787060:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19038208->19042303] (req 19038208->19042303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.787085:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19038208->19042303] (req 19038208->19042303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.787104:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19038208->19042303] (req 19038208->19042303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.787118:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962078d5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19038208->19042303] (req 19038208->19042303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760711.787202:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484155904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38049us (40298us total) trans 4294971966 rc 0/0 00000100:00100000:1.0:1789760711.787212:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4677 00000100:00100000:1.0:1789760711.787220:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701039805138 time 701080067659 state 0x2 ref 0 00000001:00080000:2.0:1789760711.787343:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971966 is committed 00000100:00100000:3.0:1789760711.790696:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760711.790886:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484156032 00000100:00100000:1.0:1789760711.790895:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484156032 00000100:00100000:1.0:1789760711.790946:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701039805138 time 701080067659 state 0x0 ref 1 00000100:00100000:1.0:1789760711.790959:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701080328075 time 701080067659 state 0x1 ref 1 00000100:00100000:1.0:1789760711.791104:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701080328075 time 701122033961 state 0x0 ref 1 00000100:00100000:1.0:1789760711.791118:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484156032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760711.791163:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19042304, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:1.0:1789760711.791171:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.791185:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:1.0:1789760711.791190:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760711.791196:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19042304->19046399] (req 19042304->19046399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.791213:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19042304->19046399] (req 19042304->19046399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760711.821930:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971967 is committed 00002000:00010000:2.0:1789760711.822105:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19042304 -> 19046400 00010000:00010000:2.0:1789760711.822120:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19042304->19046399] (req 19042304->19046399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.822134:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19042304->19046399] (req 19042304->19046399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.822143:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19042304->19046399] (req 19042304->19046399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.822150:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760711.822155:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19042304->19046399] (req 19042304->19046399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.822163:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19042304->19046399] (req 19042304->19046399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.822187:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19042304->19046399] (req 19042304->19046399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.822193:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962078dc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19042304->19046399] (req 19042304->19046399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760711.822282:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484156032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31161us (31591us total) trans 4294971967 rc 0/0 00000100:00100000:2.0:1789760711.822292:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4678 00000100:00100000:2.0:1789760711.822301:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701080328075 time 701122033961 state 0x2 ref 0 00000100:00100000:3.0:1789760711.826354:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760711.826455:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484156160 00000100:00100000:2.0:1789760711.826464:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484156160 00000100:00100000:2.0:1789760711.826518:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701080328075 time 701122033961 state 0x0 ref 1 00000100:00100000:2.0:1789760711.826534:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701122294377 time 701122033961 state 0x1 ref 1 00000100:00100000:2.0:1789760711.826573:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701122294377 time 701157503626 state 0x0 ref 1 00000100:00100000:2.0:1789760711.826586:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484156160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760711.826743:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19046400, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:2.0:1789760711.826753:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.826829:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:2.0:1789760711.826836:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760711.826842:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19046400->19050495] (req 19046400->19050495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.826857:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19046400->19050495] (req 19046400->19050495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760711.859159:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971968 is committed 00002000:00010000:2.0:1789760711.859340:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19046400 -> 19050496 00010000:00010000:2.0:1789760711.859357:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19046400->19050495] (req 19046400->19050495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.859371:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19046400->19050495] (req 19046400->19050495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.859380:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19046400->19050495] (req 19046400->19050495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.859387:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760711.859393:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19046400->19050495] (req 19046400->19050495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.859402:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19046400->19050495] (req 19046400->19050495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.859422:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19046400->19050495] (req 19046400->19050495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.859434:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078e3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19046400->19050495] (req 19046400->19050495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760711.859533:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484156160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32944us (33185us total) trans 4294971968 rc 0/0 00000100:00100000:2.0:1789760711.859552:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4679 00000100:00100000:2.0:1789760711.859562:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701122294377 time 701157503626 state 0x2 ref 0 00000100:00100000:0.0:1789760711.863534:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760711.863699:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484156288 00000100:00100000:2.0:1789760711.863710:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484156288 00000100:00100000:2.0:1789760711.863761:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701122294377 time 701157503626 state 0x0 ref 1 00000100:00100000:2.0:1789760711.863843:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701157764042 time 701157503626 state 0x1 ref 1 00000100:00100000:2.0:1789760711.863917:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701157764042 time 701194846486 state 0x0 ref 1 00000100:00100000:2.0:1789760711.863932:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484156288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760711.864052:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19050496, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:2.0:1789760711.864062:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.864122:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:2.0:1789760711.864126:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760711.864132:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19050496->19054591] (req 19050496->19054591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.864152:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19050496->19054591] (req 19050496->19054591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760711.905531:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971969 is committed 00002000:00010000:0.0:1789760711.905706:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19050496 -> 19054592 00010000:00010000:0.0:1789760711.905721:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19050496->19054591] (req 19050496->19054591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.905835:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19050496->19054591] (req 19050496->19054591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.905844:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19050496->19054591] (req 19050496->19054591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.905851:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760711.905855:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19050496->19054591] (req 19050496->19054591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.905864:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19050496->19054591] (req 19050496->19054591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.905884:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19050496->19054591] (req 19050496->19054591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760711.905891:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962078ea lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19050496->19054591] (req 19050496->19054591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760711.906109:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484156288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42175us (42582us total) trans 4294971969 rc 0/0 00000100:00100000:0.0:1789760711.906120:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4680 00000100:00100000:0.0:1789760711.906129:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701157764042 time 701194846486 state 0x2 ref 0 00000100:00100000:0.0:1789760711.912314:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760711.912433:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760711.912456:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760711.912489:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760711.912513:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760711.912560:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760711.912588:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484156416 00000100:00100000:0.0:1789760711.912598:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484156416 00000100:00100000:2.0:1789760711.912728:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484156544 00000100:00100000:2.0:1789760711.912733:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484156544 00000100:00100000:2.0:1789760711.912790:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484156800 00000100:00100000:2.0:1789760711.912793:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484156800 00000100:00100000:2.0:1789760711.912810:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484156672 00000100:00100000:2.0:1789760711.912813:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484156672 00000100:00100000:2.0:1789760711.912825:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484156928 00000100:00100000:2.0:1789760711.912839:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484156928 00000100:00100000:2.0:1789760711.912856:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 861 00000100:00100000:2.0:1789760711.912866:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484156544:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760711.912932:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484156544:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 66us (498us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.912941:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 861 00000100:00100000:2.0:1789760711.912957:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 862 00000100:00100000:2.0:1789760711.912964:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484156800:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760711.913231:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484156800:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 267us (773us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.913240:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 862 00000100:00100000:2.0:1789760711.913254:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 863 00000100:00100000:2.0:1789760711.913262:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484156672:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760711.913286:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484156672:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 26us (772us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.913293:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 863 00000100:00100000:2.0:1789760711.913305:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 864 00000100:00100000:2.0:1789760711.913311:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484156928:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760711.913412:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484156928:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 100us (851us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760711.913419:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 864 00000100:00080000:2.0:1789760711.913445:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760711.913449:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:2.0:1789760711.913465:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484157056 00000100:00100000:2.0:1789760711.913470:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484157056 00000100:00100000:2.0:1789760711.913503:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701157764042 time 701194846486 state 0x0 ref 1 00000100:00100000:2.0:1789760711.913517:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701195106902 time 701194846486 state 0x1 ref 1 00000100:00100000:2.0:1789760711.913549:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701195106902 time 701244469287 state 0x0 ref 1 00000100:00100000:2.0:1789760711.913561:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484157056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760711.913603:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19054592, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:2.0:1789760711.913611:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.913701:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:2.0:1789760711.913706:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760711.913713:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19054592->19058687] (req 19054592->19058687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.913735:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19054592->19058687] (req 19054592->19058687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760711.914428:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 467 00000100:00100000:0.0:1789760711.914442:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4d2910050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484156416:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760711.914475:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4d2910050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484156416:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 41us (1986us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760711.914482:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 467 00002000:00010000:3.0:1789760711.948650:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19054592 -> 19058688 00010000:00010000:3.0:1789760711.948668:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19054592->19058687] (req 19054592->19058687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.948680:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19054592->19058687] (req 19054592->19058687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.948688:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19054592->19058687] (req 19054592->19058687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.948695:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760711.948700:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19054592->19058687] (req 19054592->19058687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.948708:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19054592->19058687] (req 19054592->19058687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.948727:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19054592->19058687] (req 19054592->19058687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760711.948734:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962078f1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19054592->19058687] (req 19054592->19058687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760711.948819:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484157056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35256us (36510us total) trans 4294971970 rc 0/0 00000100:00100000:3.0:1789760711.948827:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4681 00000100:00100000:3.0:1789760711.948836:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701195106902 time 701244469287 state 0x2 ref 0 00000001:00080000:0.0:1789760711.949997:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971970 is committed 00000100:00100000:0.0:1789760711.952796:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760711.952861:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484157184 00000100:00100000:1.0:1789760711.952870:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484157184 00000100:00100000:1.0:1789760711.952929:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701195106902 time 701244469287 state 0x0 ref 1 00000100:00100000:1.0:1789760711.952943:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701244729703 time 701244469287 state 0x1 ref 1 00000100:00100000:1.0:1789760711.952971:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701244729703 time 701283902009 state 0x0 ref 1 00000100:00100000:1.0:1789760711.952982:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484157184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760711.953031:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19058688, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:1.0:1789760711.953040:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.953087:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:1.0:1789760711.953092:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760711.953099:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19058688->19062783] (req 19058688->19062783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760711.953114:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19058688->19062783] (req 19058688->19062783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760711.977520:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19058688 -> 19062784 00010000:00010000:2.0:1789760711.977551:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19058688->19062783] (req 19058688->19062783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.977565:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19058688->19062783] (req 19058688->19062783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.977605:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19058688->19062783] (req 19058688->19062783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.977615:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760711.977621:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19058688->19062783] (req 19058688->19062783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.977631:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19058688->19062783] (req 19058688->19062783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.977651:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19058688->19062783] (req 19058688->19062783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.977659:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962078f8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19058688->19062783] (req 19058688->19062783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760711.977720:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971971 is committed 00000100:00100000:2.0:1789760711.977757:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484157184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24771us (24966us total) trans 4294971971 rc 0/0 00000100:00100000:2.0:1789760711.977934:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4682 00000100:00100000:2.0:1789760711.977946:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701244729703 time 701283902009 state 0x2 ref 0 00000100:00100000:1.0:1789760711.979854:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760711.980045:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484157312 00000100:00100000:2.0:1789760711.980054:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484157312 00000100:00100000:2.0:1789760711.980148:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701244729703 time 701283902009 state 0x0 ref 1 00000100:00100000:2.0:1789760711.980163:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701284162425 time 701283902009 state 0x1 ref 1 00000100:00100000:2.0:1789760711.980193:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701284162425 time 701311124160 state 0x0 ref 1 00000100:00100000:2.0:1789760711.980210:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484157312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760711.980255:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19062784, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:2.0:1789760711.980264:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.980279:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:2.0:1789760711.980283:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760711.980289:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19062784->19066879] (req 19062784->19066879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760711.980307:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19062784->19066879] (req 19062784->19066879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760712.001435:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971972 is committed 00002000:00010000:3.0:1789760712.001671:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19062784 -> 19066880 00010000:00010000:3.0:1789760712.001686:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19062784->19066879] (req 19062784->19066879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.001698:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19062784->19066879] (req 19062784->19066879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.001707:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19062784->19066879] (req 19062784->19066879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.001769:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.001775:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19062784->19066879] (req 19062784->19066879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.001790:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19062784->19066879] (req 19062784->19066879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.001811:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19062784->19066879] (req 19062784->19066879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.001819:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962078ff lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19062784->19066879] (req 19062784->19066879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.002059:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484157312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 21849us (22203us total) trans 4294971972 rc 0/0 00000100:00100000:3.0:1789760712.002110:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4683 00000100:00100000:3.0:1789760712.002121:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701284162425 time 701311124160 state 0x2 ref 0 00000100:00100000:1.0:1789760712.014554:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760712.014726:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484157440 00000100:00100000:0.0:1789760712.014737:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484157440 00000100:00080000:2.0:1789760712.014808:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760712.014816:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701284162425 time 701311124160 state 0x0 ref 1 00000100:00100000:0.0:1789760712.014833:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701311384576 time 701311124160 state 0x1 ref 1 00000100:00100000:0.0:1789760712.014866:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701311384576 time 701345797542 state 0x0 ref 1 00000100:00100000:0.0:1789760712.014884:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484157440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760712.014932:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19066880, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:0.0:1789760712.014941:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.014961:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:0.0:1789760712.014964:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760712.014970:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19066880->19070975] (req 19066880->19070975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.014982:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19066880->19070975] (req 19066880->19070975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760712.070120:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971973 is committed 00002000:00010000:3.0:1789760712.070521:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19066880 -> 19070976 00010000:00010000:3.0:1789760712.070547:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19066880->19070975] (req 19066880->19070975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.070560:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19066880->19070975] (req 19066880->19070975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.070569:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19066880->19070975] (req 19066880->19070975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.070576:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.070581:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19066880->19070975] (req 19066880->19070975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.070589:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19066880->19070975] (req 19066880->19070975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.070608:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19066880->19070975] (req 19066880->19070975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.070616:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207906 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19066880->19070975] (req 19066880->19070975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.070751:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484157440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55870us (56212us total) trans 4294971973 rc 0/0 00000100:00100000:3.0:1789760712.070762:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4684 00000100:00100000:3.0:1789760712.070772:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701311384576 time 701345797542 state 0x2 ref 0 00000100:00100000:0.0:1789760712.084466:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760712.084806:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484157568 00000100:00100000:3.0:1789760712.084818:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484157568 00000100:00100000:3.0:1789760712.084873:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701311384576 time 701345797542 state 0x0 ref 1 00000100:00100000:3.0:1789760712.084888:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701346057958 time 701345797542 state 0x1 ref 1 00000100:00100000:3.0:1789760712.084920:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701346057958 time 701415851718 state 0x0 ref 1 00000100:00100000:3.0:1789760712.084933:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484157568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760712.085001:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19070976, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:3.0:1789760712.085011:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.085027:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:3.0:1789760712.085032:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760712.085039:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19070976->19075071] (req 19070976->19075071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.085053:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19070976->19075071] (req 19070976->19075071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760712.123780:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971974 is committed 00002000:00010000:3.0:1789760712.123896:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19070976 -> 19075072 00010000:00010000:3.0:1789760712.123911:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19070976->19075071] (req 19070976->19075071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.123925:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19070976->19075071] (req 19070976->19075071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.123934:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19070976->19075071] (req 19070976->19075071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.123941:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.123946:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19070976->19075071] (req 19070976->19075071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.123956:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19070976->19075071] (req 19070976->19075071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.123981:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19070976->19075071] (req 19070976->19075071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.123997:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620790d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19070976->19075071] (req 19070976->19075071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.124232:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484157568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39296us (39782us total) trans 4294971974 rc 0/0 00000100:00100000:3.0:1789760712.124243:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4685 00000100:00100000:3.0:1789760712.124253:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701346057958 time 701415851718 state 0x2 ref 0 00000100:00100000:0.0:1789760712.129828:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760712.129985:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484157696 00000100:00100000:2.0:1789760712.130057:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484157696 00000100:00100000:2.0:1789760712.130145:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701346057958 time 701415851718 state 0x0 ref 1 00000100:00100000:2.0:1789760712.130160:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701416112134 time 701415851718 state 0x1 ref 1 00000100:00100000:2.0:1789760712.130199:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701416112134 time 701461129772 state 0x0 ref 1 00000100:00100000:2.0:1789760712.130213:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484157696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760712.130262:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19075072, mtime: 0x6aad94c7, atime: 0x0, ctime: 0x6aad94c7, blocks: 0x9807 00010000:00010000:2.0:1789760712.130272:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.130295:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:2.0:1789760712.130300:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760712.130306:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19075072->19079167] (req 19075072->19079167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.131560:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19075072->19079167] (req 19075072->19079167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760712.182280:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971975 is committed 00002000:00010000:1.0:1789760712.182550:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19075072 -> 19079168 00002000:00010000:1.0:1789760712.182555:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760711 -> 1789760712 00002000:00010000:1.0:1789760712.182558:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760711 -> 1789760712 00010000:00010000:1.0:1789760712.182572:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19075072->19079167] (req 19075072->19079167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.182586:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19075072->19079167] (req 19075072->19079167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.182594:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19075072->19079167] (req 19075072->19079167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.182601:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760712.182606:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19075072->19079167] (req 19075072->19079167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.182615:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19075072->19079167] (req 19075072->19079167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.182645:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19075072->19079167] (req 19075072->19079167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.182653:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207914 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19075072->19079167] (req 19075072->19079167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760712.182753:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484157696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52538us (52929us total) trans 4294971975 rc 0/0 00000100:00100000:1.0:1789760712.182764:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4686 00000100:00100000:1.0:1789760712.182773:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701416112134 time 701461129772 state 0x2 ref 0 00000100:00100000:1.0:1789760712.186425:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760712.186555:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484157824 00000100:00100000:3.0:1789760712.186568:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484157824 00000100:00100000:3.0:1789760712.186632:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701416112134 time 701461129772 state 0x0 ref 1 00000100:00100000:3.0:1789760712.186651:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701461390188 time 701461129772 state 0x1 ref 1 00000100:00100000:3.0:1789760712.186677:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701461390188 time 701517608608 state 0x0 ref 1 00000100:00100000:3.0:1789760712.186689:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484157824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760712.186739:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19079168, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:3.0:1789760712.186748:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.186765:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:3.0:1789760712.186769:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760712.186775:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19079168->19083263] (req 19079168->19083263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.186789:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19079168->19083263] (req 19079168->19083263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760712.235992:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971976 is committed 00002000:00010000:2.0:1789760712.236150:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19079168 -> 19083264 00010000:00010000:2.0:1789760712.236166:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19079168->19083263] (req 19079168->19083263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.236191:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19079168->19083263] (req 19079168->19083263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.236201:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19079168->19083263] (req 19079168->19083263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.236209:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760712.236213:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19079168->19083263] (req 19079168->19083263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.236222:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19079168->19083263] (req 19079168->19083263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.236242:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19079168->19083263] (req 19079168->19083263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.236250:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620791b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19079168->19083263] (req 19079168->19083263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760712.236488:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484157824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49796us (50068us total) trans 4294971976 rc 0/0 00000100:00100000:2.0:1789760712.236499:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4687 00000100:00100000:2.0:1789760712.236508:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701461390188 time 701517608608 state 0x2 ref 0 00000100:00100000:2.0:1789760712.240937:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760712.241271:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484157952 00000100:00100000:3.0:1789760712.241281:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484157952 00000100:00100000:3.0:1789760712.241342:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701461390188 time 701517608608 state 0x0 ref 1 00000100:00100000:3.0:1789760712.241358:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701517869024 time 701517608608 state 0x1 ref 1 00000100:00100000:3.0:1789760712.241385:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701517869024 time 701572316647 state 0x0 ref 1 00000100:00100000:3.0:1789760712.241398:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484157952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760712.241443:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19083264, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:3.0:1789760712.241451:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.241467:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:3.0:1789760712.241472:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760712.241479:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19083264->19087359] (req 19083264->19087359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.241493:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19083264->19087359] (req 19083264->19087359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760712.291336:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19083264 -> 19087360 00010000:00010000:3.0:1789760712.291355:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19083264->19087359] (req 19083264->19087359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.291369:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19083264->19087359] (req 19083264->19087359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.291378:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19083264->19087359] (req 19083264->19087359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.291385:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.291390:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19083264->19087359] (req 19083264->19087359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.291399:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19083264->19087359] (req 19083264->19087359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.291418:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19083264->19087359] (req 19083264->19087359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.291425:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207922 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19083264->19087359] (req 19083264->19087359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.291703:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484157952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50302us (50771us total) trans 4294971977 rc 0/0 00000100:00100000:3.0:1789760712.291717:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4688 00000100:00100000:3.0:1789760712.291727:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701517869024 time 701572316647 state 0x2 ref 0 00000001:00080000:3.0:1789760712.291765:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971977 is committed 00000100:00100000:0.0:1789760712.298055:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760712.298392:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484158080 00000100:00100000:3.0:1789760712.298401:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484158080 00000100:00100000:3.0:1789760712.298523:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701517869024 time 701572316647 state 0x0 ref 1 00000100:00100000:3.0:1789760712.298728:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701572577063 time 701572316647 state 0x1 ref 1 00000100:00100000:3.0:1789760712.298808:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701572577063 time 701629737930 state 0x0 ref 1 00000100:00100000:3.0:1789760712.298822:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484158080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760712.298873:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19087360, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:3.0:1789760712.298882:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.298898:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:3.0:1789760712.298903:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760712.298908:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19087360->19091455] (req 19087360->19091455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.298922:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19087360->19091455] (req 19087360->19091455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760712.332507:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19087360 -> 19091456 00010000:00010000:3.0:1789760712.332529:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19087360->19091455] (req 19087360->19091455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.332554:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19087360->19091455] (req 19087360->19091455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.332563:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19087360->19091455] (req 19087360->19091455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.332570:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.333754:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19087360->19091455] (req 19087360->19091455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.333770:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19087360->19091455] (req 19087360->19091455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.333792:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19087360->19091455] (req 19087360->19091455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.333801:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207929 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19087360->19091455] (req 19087360->19091455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.333939:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484158080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35117us (35891us total) trans 4294971978 rc 0/0 00000100:00100000:3.0:1789760712.333950:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4689 00000100:00100000:3.0:1789760712.333960:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701572577063 time 701629737930 state 0x2 ref 0 00000001:00080000:3.0:1789760712.334006:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971978 is committed 00000100:00100000:1.0:1789760712.338372:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760712.339161:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484158208 00000100:00100000:2.0:1789760712.339273:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484158208 00000100:00100000:2.0:1789760712.339341:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701572577063 time 701629737930 state 0x0 ref 1 00000100:00100000:2.0:1789760712.339358:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701629998346 time 701629737930 state 0x1 ref 1 00000100:00100000:2.0:1789760712.339391:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701629998346 time 701670318682 state 0x0 ref 1 00000100:00100000:2.0:1789760712.339406:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484158208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760712.339460:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19091456, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:2.0:1789760712.339470:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.339486:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:2.0:1789760712.339490:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760712.339496:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19091456->19095551] (req 19091456->19095551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.339510:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19091456->19095551] (req 19091456->19095551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760712.400331:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19091456 -> 19095552 00010000:00010000:3.0:1789760712.400354:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19091456->19095551] (req 19091456->19095551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.400368:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19091456->19095551] (req 19091456->19095551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.400377:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19091456->19095551] (req 19091456->19095551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.400385:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.400390:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19091456->19095551] (req 19091456->19095551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.400399:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19091456->19095551] (req 19091456->19095551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.400419:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19091456->19095551] (req 19091456->19095551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.400427:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207930 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19091456->19095551] (req 19091456->19095551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.400518:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484158208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61110us (62153us total) trans 4294971979 rc 0/0 00000100:00100000:3.0:1789760712.400546:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4690 00000100:00100000:3.0:1789760712.400558:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701629998346 time 701670318682 state 0x2 ref 0 00000001:00080000:3.0:1789760712.400607:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971979 is committed 00000100:00100000:2.0:1789760712.407226:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760712.407461:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484158336 00000100:00100000:3.0:1789760712.407471:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484158336 00000100:00100000:3.0:1789760712.407530:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701629998346 time 701670318682 state 0x0 ref 1 00000100:00100000:3.0:1789760712.407554:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701670579098 time 701670318682 state 0x1 ref 1 00000100:00100000:3.0:1789760712.407578:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701670579098 time 701738509176 state 0x0 ref 1 00000100:00100000:3.0:1789760712.407591:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484158336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760712.407655:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19095552, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:3.0:1789760712.407664:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.407682:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:3.0:1789760712.407686:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760712.407692:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19095552->19099647] (req 19095552->19099647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.407707:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19095552->19099647] (req 19095552->19099647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760712.486587:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971980 is committed 00002000:00010000:3.0:1789760712.486784:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19095552 -> 19099648 00010000:00010000:3.0:1789760712.486799:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19095552->19099647] (req 19095552->19099647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.486811:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19095552->19099647] (req 19095552->19099647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.486820:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19095552->19099647] (req 19095552->19099647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.486827:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.486831:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19095552->19099647] (req 19095552->19099647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.486839:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19095552->19099647] (req 19095552->19099647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.486857:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19095552->19099647] (req 19095552->19099647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.486864:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207937 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19095552->19099647] (req 19095552->19099647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.486952:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484158336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79359us (79730us total) trans 4294971980 rc 0/0 00000100:00100000:3.0:1789760712.486962:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4691 00000100:00100000:3.0:1789760712.486973:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701670579098 time 701738509176 state 0x2 ref 0 00000100:00100000:0.0:1789760712.492497:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760712.492709:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484158464 00000100:00100000:3.0:1789760712.492719:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484158464 00000100:00100000:3.0:1789760712.492772:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701670579098 time 701738509176 state 0x0 ref 1 00000100:00100000:3.0:1789760712.492787:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701738769592 time 701738509176 state 0x1 ref 1 00000100:00100000:3.0:1789760712.492818:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701738769592 time 701823748996 state 0x0 ref 1 00000100:00100000:3.0:1789760712.492831:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484158464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760712.492879:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19099648, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:3.0:1789760712.492888:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.492905:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:3.0:1789760712.492910:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760712.492916:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19099648->19103743] (req 19099648->19103743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.492930:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19099648->19103743] (req 19099648->19103743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760712.537219:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19099648 -> 19103744 00010000:00010000:0.0:1789760712.537239:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19099648->19103743] (req 19099648->19103743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.537253:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19099648->19103743] (req 19099648->19103743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.537262:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19099648->19103743] (req 19099648->19103743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.537275:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760712.537280:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19099648->19103743] (req 19099648->19103743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.537290:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19099648->19103743] (req 19099648->19103743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.537309:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19099648->19103743] (req 19099648->19103743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.537330:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620793e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19099648->19103743] (req 19099648->19103743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760712.537423:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484158464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44589us (44931us total) trans 4294971981 rc 0/0 00000100:00100000:0.0:1789760712.537433:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4692 00000100:00100000:0.0:1789760712.537442:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701738769592 time 701823748996 state 0x2 ref 0 00000001:00080000:2.0:1789760712.538271:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971981 is committed 00000100:00100000:3.0:1789760712.540461:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760712.540726:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484158592 00000100:00100000:0.0:1789760712.540736:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484158592 00000100:00100000:0.0:1789760712.540795:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701738769592 time 701823748996 state 0x0 ref 1 00000100:00100000:0.0:1789760712.540809:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701824009412 time 701823748996 state 0x1 ref 1 00000100:00100000:0.0:1789760712.540838:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701824009412 time 701871769059 state 0x0 ref 1 00000100:00100000:0.0:1789760712.540854:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484158592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760712.540899:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19103744, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:0.0:1789760712.540908:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.540923:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:0.0:1789760712.540928:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760712.540934:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19103744->19107839] (req 19103744->19107839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.540947:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19103744->19107839] (req 19103744->19107839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760712.573940:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971982 is committed 00002000:00010000:0.0:1789760712.574155:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19103744 -> 19107840 00010000:00010000:0.0:1789760712.574170:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19103744->19107839] (req 19103744->19107839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.574183:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19103744->19107839] (req 19103744->19107839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.574192:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19103744->19107839] (req 19103744->19107839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.574199:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760712.574203:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19103744->19107839] (req 19103744->19107839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.574212:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19103744->19107839] (req 19103744->19107839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.574232:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19103744->19107839] (req 19103744->19107839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.574239:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207945 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19103744->19107839] (req 19103744->19107839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760712.574427:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484158592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33570us (33968us total) trans 4294971982 rc 0/0 00000100:00100000:0.0:1789760712.574438:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4693 00000100:00100000:0.0:1789760712.574448:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701824009412 time 701871769059 state 0x2 ref 0 00000100:00100000:2.0:1789760712.578298:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760712.578437:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484158720 00000100:00100000:3.0:1789760712.578447:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484158720 00000100:00100000:3.0:1789760712.578499:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701824009412 time 701871769059 state 0x0 ref 1 00000100:00100000:3.0:1789760712.578518:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701872029475 time 701871769059 state 0x1 ref 1 00000100:00100000:3.0:1789760712.578681:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701872029475 time 701909550076 state 0x0 ref 1 00000100:00100000:3.0:1789760712.578698:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484158720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760712.578747:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19107840, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:3.0:1789760712.578756:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.578771:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:3.0:1789760712.578775:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760712.578782:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19107840->19111935] (req 19107840->19111935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.578795:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19107840->19111935] (req 19107840->19111935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.589303:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760712.589953:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508908288:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1789760712.590022:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760712.590105:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508908288 00000100:00100000:2.0:1789760712.590112:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508908288 00000100:00100000:2.0:1789760712.590152:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 358 00000100:00100000:2.0:1789760712.590160:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb727b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508908288:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:2.0:1789760712.590180:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528416768 pending 1703936 free 3838836736 avail 3817865216 00000100:00100000:2.0:1789760712.590218:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb727b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508908288:12345-0@lo:13:osp-pre-0-0.0 Request processed in 58us (196us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760712.590224:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 358 00000100:00100000:0.0:1789760712.590912:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b18000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508908544:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760712.590949:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760712.590994:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508908544 00000100:00100000:0.0:1789760712.590999:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508908544 00000100:00100000:0.0:1789760712.591031:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 417 00000100:00100000:0.0:1789760712.591039:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b18700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508908544:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:0.0:1789760712.591090:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:0.0:1789760712.591129:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508908544:12345-0@lo:41:osp-pre-0-1.0 Request processed in 91us (178us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760712.591135:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 417 00000100:00100000:0.0:1789760712.591176:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b18000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508908544:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760712.591198:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df02ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508908416:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760712.591223:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760712.591256:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508908416 00000100:00100000:2.0:1789760712.591563:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508908288:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760712.591946:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508908416 00000100:00100000:0.0:1789760712.591985:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 359 00000100:00100000:0.0:1789760712.591992:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b18000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508908416:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:0.0:1789760712.592008:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528416768 pending 1703936 free 3838836736 avail 3817865216 00000100:00100000:0.0:1789760712.592051:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508908416:12345-0@lo:13:osp-pre-0-1.0 Request processed in 59us (826us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760712.592058:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 359 00000100:00100000:0.0:1789760712.592114:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df02ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508908416:0@lo:13:osp-pre-0-1.0 00000001:00080000:0.0:1789760712.630511:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971983 is committed 00002000:00010000:3.0:1789760712.630829:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19107840 -> 19111936 00010000:00010000:3.0:1789760712.630845:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19107840->19111935] (req 19107840->19111935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.630858:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19107840->19111935] (req 19107840->19111935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.630867:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19107840->19111935] (req 19107840->19111935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.630874:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.630879:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19107840->19111935] (req 19107840->19111935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.630888:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19107840->19111935] (req 19107840->19111935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.630906:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19107840->19111935] (req 19107840->19111935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.630913:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620794c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19107840->19111935] (req 19107840->19111935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.635775:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484158720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54616us (55021us total) trans 4294971983 rc 0/0 00000100:00100000:3.0:1789760712.635792:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4694 00000100:00100000:3.0:1789760712.635802:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701872029475 time 701909550076 state 0x2 ref 0 00000100:00100000:2.0:1789760712.650420:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760712.650563:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484158848 00000100:00100000:3.0:1789760712.650574:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484158848 00000100:00100000:3.0:1789760712.650640:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701872029475 time 701909550076 state 0x0 ref 1 00000100:00100000:3.0:1789760712.650656:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701909810492 time 701909550076 state 0x1 ref 1 00000100:00100000:3.0:1789760712.650686:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701909810492 time 701981616870 state 0x0 ref 1 00000100:00100000:3.0:1789760712.650703:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484158848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760712.650752:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19111936, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:3.0:1789760712.650761:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.650777:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=701s v=(0 0) 00010000:00010000:3.0:1789760712.650782:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760712.650789:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19111936->19116031] (req 19111936->19116031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.650804:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19111936->19116031] (req 19111936->19116031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760712.697485:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19111936 -> 19116032 00010000:00010000:3.0:1789760712.697505:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19111936->19116031] (req 19111936->19116031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.697518:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19111936->19116031] (req 19111936->19116031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.697527:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19111936->19116031] (req 19111936->19116031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.697546:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.697551:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19111936->19116031] (req 19111936->19116031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.697560:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19111936->19116031] (req 19111936->19116031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.697581:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19111936->19116031] (req 19111936->19116031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.697589:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207953 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19111936->19116031] (req 19111936->19116031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.697697:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484158848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46992us (47279us total) trans 4294971984 rc 0/0 00000100:00100000:3.0:1789760712.697707:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4695 00000100:00100000:3.0:1789760712.697718:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701909810492 time 701981616870 state 0x2 ref 0 00000100:00100000:3.0:1789760712.701566:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760712.701664:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484158976 00000100:00100000:2.0:1789760712.701674:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484158976 00000100:00100000:2.0:1789760712.701726:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701909810492 time 701981616870 state 0x0 ref 1 00000100:00100000:2.0:1789760712.701741:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701981877286 time 701981616870 state 0x1 ref 1 00000100:00100000:2.0:1789760712.701769:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701981877286 time 702032700391 state 0x0 ref 1 00000100:00100000:2.0:1789760712.701782:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484158976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760712.701829:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19116032, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:2.0:1789760712.701838:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.701853:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:2.0:1789760712.701857:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760712.701863:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19116032->19120127] (req 19116032->19120127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.701881:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19116032->19120127] (req 19116032->19120127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760712.703054:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971984 is committed 00000001:00080000:3.0:1789760712.745837:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971985 is committed 00002000:00010000:2.0:1789760712.746340:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19116032 -> 19120128 00010000:00010000:2.0:1789760712.746356:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19116032->19120127] (req 19116032->19120127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.746369:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19116032->19120127] (req 19116032->19120127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.746379:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19116032->19120127] (req 19116032->19120127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.746387:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760712.746392:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19116032->19120127] (req 19116032->19120127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.746403:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19116032->19120127] (req 19116032->19120127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.746423:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19116032->19120127] (req 19116032->19120127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.746432:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620795a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19116032->19120127] (req 19116032->19120127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760712.746541:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484158976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44750us (44972us total) trans 4294971985 rc 0/0 00000100:00100000:2.0:1789760712.746552:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4696 00000100:00100000:2.0:1789760712.746562:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701981877286 time 702032700391 state 0x2 ref 0 00000100:00100000:2.0:1789760712.752252:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760712.752339:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484159104 00000100:00100000:0.0:1789760712.752349:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484159104 00000100:00100000:0.0:1789760712.752413:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 701981877286 time 702032700391 state 0x0 ref 1 00000100:00100000:0.0:1789760712.752428:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702032960807 time 702032700391 state 0x1 ref 1 00000100:00100000:0.0:1789760712.752457:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702032960807 time 702083388740 state 0x0 ref 1 00000100:00100000:0.0:1789760712.752479:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484159104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760712.752524:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19120128, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:0.0:1789760712.752543:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.752563:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:0.0:1789760712.752567:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760712.752582:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19120128->19124223] (req 19120128->19124223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.752594:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19120128->19124223] (req 19120128->19124223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760712.796589:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19120128 -> 19124224 00010000:00010000:1.0:1789760712.796610:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19120128->19124223] (req 19120128->19124223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.796623:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19120128->19124223] (req 19120128->19124223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.796631:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19120128->19124223] (req 19120128->19124223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.796638:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760712.796643:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19120128->19124223] (req 19120128->19124223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.796652:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19120128->19124223] (req 19120128->19124223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.796671:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19120128->19124223] (req 19120128->19124223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760712.796852:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971986 is committed 00010000:00010000:1.0:1789760712.796931:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207961 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19120128->19124223] (req 19120128->19124223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760712.797020:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484159104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44547us (44772us total) trans 4294971986 rc 0/0 00000100:00100000:1.0:1789760712.797030:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4697 00000100:00100000:1.0:1789760712.797039:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702032960807 time 702083388740 state 0x2 ref 0 00000100:00100000:3.0:1789760712.800153:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760712.800285:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484159232 00000100:00100000:0.0:1789760712.800296:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484159232 00000100:00100000:0.0:1789760712.800419:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702032960807 time 702083388740 state 0x0 ref 1 00000100:00100000:0.0:1789760712.800435:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702083649156 time 702083388740 state 0x1 ref 1 00000100:00100000:0.0:1789760712.800468:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702083649156 time 702131398449 state 0x0 ref 1 00000100:00100000:0.0:1789760712.800481:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484159232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760712.800523:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19124224, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:0.0:1789760712.800723:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.800739:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:0.0:1789760712.800744:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760712.800750:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19124224->19128319] (req 19124224->19128319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.800769:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19124224->19128319] (req 19124224->19128319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760712.840097:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971987 is committed 00002000:00010000:1.0:1789760712.840638:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19124224 -> 19128320 00010000:00010000:1.0:1789760712.840654:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19124224->19128319] (req 19124224->19128319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.840667:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19124224->19128319] (req 19124224->19128319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.840677:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19124224->19128319] (req 19124224->19128319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.840684:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760712.840689:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19124224->19128319] (req 19124224->19128319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.840699:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19124224->19128319] (req 19124224->19128319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.840725:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19124224->19128319] (req 19124224->19128319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.840734:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207968 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19124224->19128319] (req 19124224->19128319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760712.840830:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484159232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40346us (40682us total) trans 4294971987 rc 0/0 00000100:00100000:1.0:1789760712.840840:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4698 00000100:00100000:1.0:1789760712.840851:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702083649156 time 702131398449 state 0x2 ref 0 00000100:00100000:3.0:1789760712.846688:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760712.848929:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484159360 00000100:00100000:2.0:1789760712.848944:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484159360 00000100:00100000:2.0:1789760712.849002:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702083649156 time 702131398449 state 0x0 ref 1 00000100:00100000:2.0:1789760712.849017:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702131658865 time 702131398449 state 0x1 ref 1 00000100:00100000:2.0:1789760712.849207:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702131658865 time 702180136550 state 0x0 ref 1 00000100:00100000:2.0:1789760712.849221:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7070050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484159360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760712.849268:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19128320, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:2.0:1789760712.849276:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.849292:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:2.0:1789760712.849296:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760712.849303:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19128320->19132415] (req 19128320->19132415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.849332:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19128320->19132415] (req 19128320->19132415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760712.885264:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971988 is committed 00002000:00010000:2.0:1789760712.885655:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19128320 -> 19132416 00010000:00010000:2.0:1789760712.885670:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19128320->19132415] (req 19128320->19132415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.885683:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19128320->19132415] (req 19128320->19132415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.885692:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19128320->19132415] (req 19128320->19132415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.885700:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760712.885704:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19128320->19132415] (req 19128320->19132415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.885715:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19128320->19132415] (req 19128320->19132415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.885737:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19128320->19132415] (req 19128320->19132415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760712.885745:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620796f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19128320->19132415] (req 19128320->19132415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760712.885842:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7070050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484159360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36618us (39158us total) trans 4294971988 rc 0/0 00000100:00100000:2.0:1789760712.885853:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4699 00000100:00100000:2.0:1789760712.885863:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702131658865 time 702180136550 state 0x2 ref 0 00000100:00100000:2.0:1789760712.905347:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760712.905456:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484159488 00000100:00100000:0.0:1789760712.905465:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484159488 00000100:00100000:0.0:1789760712.905525:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702131658865 time 702180136550 state 0x0 ref 1 00000100:00100000:0.0:1789760712.905549:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702180396966 time 702180136550 state 0x1 ref 1 00000100:00100000:0.0:1789760712.905582:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702180396966 time 702236512850 state 0x0 ref 1 00000100:00100000:0.0:1789760712.905596:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484159488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760712.905713:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19132416, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:0.0:1789760712.905722:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.905736:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:0.0:1789760712.905741:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760712.905747:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19132416->19136511] (req 19132416->19136511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760712.905760:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19132416->19136511] (req 19132416->19136511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760712.987285:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971989 is committed 00002000:00010000:3.0:1789760712.987633:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19132416 -> 19136512 00010000:00010000:3.0:1789760712.987654:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19132416->19136511] (req 19132416->19136511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.987670:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19132416->19136511] (req 19132416->19136511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.987681:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19132416->19136511] (req 19132416->19136511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.987691:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760712.987698:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19132416->19136511] (req 19132416->19136511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.987709:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19132416->19136511] (req 19132416->19136511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.987736:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19132416->19136511] (req 19132416->19136511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760712.987744:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207976 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19132416->19136511] (req 19132416->19136511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760712.987883:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484159488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 82283us (82539us total) trans 4294971989 rc 0/0 00000100:00100000:3.0:1789760712.987896:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4700 00000100:00100000:3.0:1789760712.987908:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702180396966 time 702236512850 state 0x2 ref 0 00000100:00100000:3.0:1789760712.994730:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760712.998550:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484159616 00000100:00100000:1.0:1789760712.998570:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484159616 00000100:00100000:1.0:1789760712.998660:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702180396966 time 702236512850 state 0x0 ref 1 00000100:00100000:1.0:1789760712.998679:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702236773266 time 702236512850 state 0x1 ref 1 00000100:00100000:1.0:1789760712.998718:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702236773266 time 702329648642 state 0x0 ref 1 00000100:00100000:1.0:1789760712.998734:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484159616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760712.998799:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19136512, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:1.0:1789760712.998809:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.998836:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:1.0:1789760712.998841:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760712.998849:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19136512->19140607] (req 19136512->19140607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760712.998865:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19136512->19140607] (req 19136512->19140607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760713.054692:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19136512 -> 19140608 00010000:00010000:1.0:1789760713.054718:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19136512->19140607] (req 19136512->19140607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.054735:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19136512->19140607] (req 19136512->19140607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.054746:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19136512->19140607] (req 19136512->19140607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.054754:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760713.054760:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19136512->19140607] (req 19136512->19140607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.054772:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19136512->19140607] (req 19136512->19140607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.054808:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19136512->19140607] (req 19136512->19140607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.054816:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620797d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19136512->19140607] (req 19136512->19140607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760713.054930:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484159616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56194us (60208us total) trans 4294971990 rc 0/0 00000100:00100000:1.0:1789760713.054942:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4701 00000100:00100000:1.0:1789760713.054955:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702236773266 time 702329648642 state 0x2 ref 0 00000001:00080000:1.0:1789760713.055001:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971990 is committed 00000100:00100000:1.0:1789760713.059909:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760713.060022:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484159744 00000100:00100000:2.0:1789760713.060034:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484159744 00000100:00100000:2.0:1789760713.060283:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702236773266 time 702329648642 state 0x0 ref 1 00000100:00100000:2.0:1789760713.060303:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702329909058 time 702329648642 state 0x1 ref 1 00000100:00100000:2.0:1789760713.060498:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702329909058 time 702391426925 state 0x0 ref 1 00000100:00100000:2.0:1789760713.060515:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484159744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760713.060593:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19140608, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:2.0:1789760713.060604:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.060695:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:2.0:1789760713.060700:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760713.060708:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19140608->19144703] (req 19140608->19144703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.060725:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19140608->19144703] (req 19140608->19144703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760713.104188:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19140608 -> 19144704 00010000:00010000:2.0:1789760713.104214:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19140608->19144703] (req 19140608->19144703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.104231:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19140608->19144703] (req 19140608->19144703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.104241:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19140608->19144703] (req 19140608->19144703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.104249:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760713.104256:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19140608->19144703] (req 19140608->19144703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.104266:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19140608->19144703] (req 19140608->19144703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.104290:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19140608->19144703] (req 19140608->19144703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.104298:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207984 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19140608->19144703] (req 19140608->19144703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760713.104535:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484159744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44009us (44625us total) trans 4294971991 rc 0/0 00000100:00100000:2.0:1789760713.104553:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4702 00000100:00100000:2.0:1789760713.104566:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702329909058 time 702391426925 state 0x2 ref 0 00000001:00080000:0.0:1789760713.105408:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971991 is committed 00000100:00100000:2.0:1789760713.110630:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760713.110966:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484159872 00000100:00100000:0.0:1789760713.110980:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484159872 00000100:00100000:0.0:1789760713.111157:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702329909058 time 702391426925 state 0x0 ref 1 00000100:00100000:0.0:1789760713.111250:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702391687341 time 702391426925 state 0x1 ref 1 00000100:00100000:0.0:1789760713.111288:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702391687341 time 702442218683 state 0x0 ref 1 00000100:00100000:0.0:1789760713.111304:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484159872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760713.112472:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19144704, mtime: 0x6aad94c8, atime: 0x0, ctime: 0x6aad94c8, blocks: 0x9807 00010000:00010000:0.0:1789760713.112487:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.112509:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:0.0:1789760713.112514:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760713.112523:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19144704->19148799] (req 19144704->19148799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.112550:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19144704->19148799] (req 19144704->19148799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760713.112893:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1789760713.112917:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4defdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508908672:0@lo:41:osp-pre-1-0.0 00000100:00100000:3.0:1789760713.112998:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760713.113303:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508908672 00000100:00100000:1.0:1789760713.113320:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508908672 00000100:00100000:1.0:1789760713.113444:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 418 00000100:00100000:1.0:1789760713.113454:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810ad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508908672:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760713.113502:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760713.113571:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810ad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508908672:12345-0@lo:41:osp-pre-1-0.0 Request processed in 117us (571us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760713.113580:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 418 00000100:00100000:3.0:1789760713.114248:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4defdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508908672:0@lo:41:osp-pre-1-0.0 00000001:00080000:3.0:1789760713.216914:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971992 is committed 00002000:00010000:0.0:1789760713.217747:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19144704 -> 19148800 00002000:00010000:0.0:1789760713.217754:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760712 -> 1789760713 00002000:00010000:0.0:1789760713.217758:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760712 -> 1789760713 00010000:00010000:0.0:1789760713.217774:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19144704->19148799] (req 19144704->19148799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.217790:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19144704->19148799] (req 19144704->19148799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.217864:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19144704->19148799] (req 19144704->19148799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.217873:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760713.217879:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19144704->19148799] (req 19144704->19148799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.217891:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19144704->19148799] (req 19144704->19148799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.217917:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19144704->19148799] (req 19144704->19148799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.217926:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620798b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19144704->19148799] (req 19144704->19148799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760713.218369:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484159872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 107061us (107757us total) trans 4294971992 rc 0/0 00000100:00100000:0.0:1789760713.218382:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4703 00000100:00100000:0.0:1789760713.218394:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702391687341 time 702442218683 state 0x2 ref 0 00000100:00100000:3.0:1789760713.221634:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760713.222022:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484160000 00000100:00100000:0.0:1789760713.222047:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484160000 00000100:00100000:0.0:1789760713.222155:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702391687341 time 702442218683 state 0x0 ref 1 00000100:00100000:0.0:1789760713.222190:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702442479099 time 702442218683 state 0x1 ref 1 00000100:00100000:0.0:1789760713.222221:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702442479099 time 702553152155 state 0x0 ref 1 00000100:00100000:0.0:1789760713.222238:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484160000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760713.222300:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19148800, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:0.0:1789760713.222322:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.222343:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:0.0:1789760713.222349:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760713.222356:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19148800->19152895] (req 19148800->19152895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.222374:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19148800->19152895] (req 19148800->19152895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760713.332759:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19148800 -> 19152896 00010000:00010000:3.0:1789760713.332794:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19148800->19152895] (req 19148800->19152895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.332812:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19148800->19152895] (req 19148800->19152895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.332822:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19148800->19152895] (req 19148800->19152895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.332831:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760713.332837:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19148800->19152895] (req 19148800->19152895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.332848:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19148800->19152895] (req 19148800->19152895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.332875:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19148800->19152895] (req 19148800->19152895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.332883:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207992 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19148800->19152895] (req 19148800->19152895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760713.333029:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484160000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 110788us (111412us total) trans 4294971993 rc 0/0 00000100:00100000:3.0:1789760713.333043:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4704 00000100:00100000:3.0:1789760713.333055:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702442479099 time 702553152155 state 0x2 ref 0 00000001:00080000:1.0:1789760713.334425:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971993 is committed 00000100:00100000:3.0:1789760713.338673:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760713.339327:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484160128 00000100:00100000:1.0:1789760713.339341:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484160128 00000100:00100000:1.0:1789760713.339416:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702442479099 time 702553152155 state 0x0 ref 1 00000100:00100000:1.0:1789760713.339435:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702553412571 time 702553152155 state 0x1 ref 1 00000100:00100000:1.0:1789760713.339469:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702553412571 time 702670399477 state 0x0 ref 1 00000100:00100000:1.0:1789760713.339484:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484160128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760713.339562:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19152896, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:1.0:1789760713.339573:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.339593:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:1.0:1789760713.339598:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760713.339606:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19152896->19156991] (req 19152896->19156991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.340728:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19152896->19156991] (req 19152896->19156991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760713.390496:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971994 is committed 00002000:00010000:0.0:1789760713.390612:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19152896 -> 19156992 00010000:00010000:0.0:1789760713.390632:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19152896->19156991] (req 19152896->19156991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.390655:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19152896->19156991] (req 19152896->19156991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.390664:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19152896->19156991] (req 19152896->19156991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.390672:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760713.390677:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19152896->19156991] (req 19152896->19156991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.390687:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19152896->19156991] (req 19152896->19156991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.390715:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19152896->19156991] (req 19152896->19156991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.390724:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207999 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19152896->19156991] (req 19152896->19156991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760713.390862:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484160128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51374us (52198us total) trans 4294971994 rc 0/0 00000100:00100000:0.0:1789760713.390873:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4705 00000100:00100000:0.0:1789760713.390885:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702553412571 time 702670399477 state 0x2 ref 0 00000100:00100000:2.0:1789760713.406257:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760713.406735:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484160256 00000100:00100000:0.0:1789760713.406751:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484160256 00000100:00100000:0.0:1789760713.407194:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702553412571 time 702670399477 state 0x0 ref 1 00000100:00100000:0.0:1789760713.407214:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702670659893 time 702670399477 state 0x1 ref 1 00000100:00100000:0.0:1789760713.407372:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702670659893 time 702738240635 state 0x0 ref 1 00000100:00100000:0.0:1789760713.407392:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484160256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760713.407603:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19156992, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:0.0:1789760713.407617:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.407636:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:0.0:1789760713.407641:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760713.407650:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19156992->19161087] (req 19156992->19161087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.407669:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19156992->19161087] (req 19156992->19161087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760713.469205:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971995 is committed 00002000:00010000:1.0:1789760713.469342:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19156992 -> 19161088 00010000:00010000:1.0:1789760713.469363:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19156992->19161087] (req 19156992->19161087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.469379:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19156992->19161087] (req 19156992->19161087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.469389:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19156992->19161087] (req 19156992->19161087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.469397:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760713.469402:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19156992->19161087] (req 19156992->19161087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.469415:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19156992->19161087] (req 19156992->19161087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.469439:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19156992->19161087] (req 19156992->19161087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.469447:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962079a0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19156992->19161087] (req 19156992->19161087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760713.470332:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484160256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62933us (64086us total) trans 4294971995 rc 0/0 00000100:00100000:1.0:1789760713.470349:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4706 00000100:00100000:1.0:1789760713.470366:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702670659893 time 702738240635 state 0x2 ref 0 00000100:00100000:0.0:1789760713.477458:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760713.477621:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484160384 00000100:00100000:3.0:1789760713.477637:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484160384 00000100:00100000:3.0:1789760713.477714:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702670659893 time 702738240635 state 0x0 ref 1 00000100:00100000:3.0:1789760713.477733:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702738501051 time 702738240635 state 0x1 ref 1 00000100:00100000:3.0:1789760713.477775:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702738501051 time 702808705778 state 0x0 ref 1 00000100:00100000:3.0:1789760713.477790:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484160384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760713.477856:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19161088, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:3.0:1789760713.477868:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.477887:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:3.0:1789760713.477892:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760713.477901:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19161088->19165183] (req 19161088->19165183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.477917:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19161088->19165183] (req 19161088->19165183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760713.529118:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971996 is committed 00002000:00010000:0.0:1789760713.529401:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19161088 -> 19165184 00010000:00010000:0.0:1789760713.529421:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19161088->19165183] (req 19161088->19165183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.529436:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19161088->19165183] (req 19161088->19165183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.529448:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19161088->19165183] (req 19161088->19165183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.529456:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760713.529463:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19161088->19165183] (req 19161088->19165183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.529474:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19161088->19165183] (req 19161088->19165183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.529500:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19161088->19165183] (req 19161088->19165183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.529509:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962079a7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19161088->19165183] (req 19161088->19165183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760713.529862:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484160384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52068us (52412us total) trans 4294971996 rc 0/0 00000100:00100000:0.0:1789760713.529876:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4707 00000100:00100000:0.0:1789760713.529889:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702738501051 time 702808705778 state 0x2 ref 0 00000100:00100000:3.0:1789760713.537787:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760713.537899:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484160512 00000100:00100000:0.0:1789760713.537912:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484160512 00000100:00100000:0.0:1789760713.537988:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702738501051 time 702808705778 state 0x0 ref 1 00000100:00100000:0.0:1789760713.538008:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702808966194 time 702808705778 state 0x1 ref 1 00000100:00100000:0.0:1789760713.538044:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702808966194 time 702868975116 state 0x0 ref 1 00000100:00100000:0.0:1789760713.538060:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484160512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760713.539187:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19165184, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:0.0:1789760713.539264:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.539285:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:0.0:1789760713.539290:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760713.539298:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19165184->19169279] (req 19165184->19169279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.539380:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19165184->19169279] (req 19165184->19169279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.618097:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760713.618491:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971997 is committed 00002000:00010000:3.0:1789760713.618756:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19165184 -> 19169280 00010000:00010000:3.0:1789760713.618776:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19165184->19169279] (req 19165184->19169279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.618791:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19165184->19169279] (req 19165184->19169279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.618807:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19165184->19169279] (req 19165184->19169279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.618815:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760713.618821:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19165184->19169279] (req 19165184->19169279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.618907:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19165184->19169279] (req 19165184->19169279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.618938:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19165184->19169279] (req 19165184->19169279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.618947:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962079ae lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19165184->19169279] (req 19165184->19169279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760713.619299:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484160512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 81234us (81518us total) trans 4294971997 rc 0/0 00000100:00100000:3.0:1789760713.619320:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4708 00000100:00100000:3.0:1789760713.619334:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702808966194 time 702868975116 state 0x2 ref 0 00000100:00100000:2.0:1789760713.628037:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760713.628173:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484160640 00000100:00100000:0.0:1789760713.628186:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484160640 00000100:00100000:0.0:1789760713.628281:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702808966194 time 702868975116 state 0x0 ref 1 00000100:00100000:0.0:1789760713.628300:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702869235532 time 702868975116 state 0x1 ref 1 00000100:00100000:0.0:1789760713.628343:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702869235532 time 702959273164 state 0x0 ref 1 00000100:00100000:0.0:1789760713.628360:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484160640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760713.628424:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19169280, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:0.0:1789760713.628433:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.628449:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=702s v=(0 0) 00010000:00010000:0.0:1789760713.628453:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760713.628459:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19169280->19173375] (req 19169280->19173375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760713.628471:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19169280->19173375] (req 19169280->19173375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760713.688208:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19169280 -> 19173376 00010000:00010000:1.0:1789760713.688238:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19169280->19173375] (req 19169280->19173375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.688255:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19169280->19173375] (req 19169280->19173375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.688266:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19169280->19173375] (req 19169280->19173375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.688280:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760713.688285:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19169280->19173375] (req 19169280->19173375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.688295:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19169280->19173375] (req 19169280->19173375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.688953:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19169280->19173375] (req 19169280->19173375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.688967:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962079b5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19169280->19173375] (req 19169280->19173375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760713.689164:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484160640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60798us (61130us total) trans 4294971998 rc 0/0 00000100:00100000:1.0:1789760713.689179:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4709 00000100:00100000:1.0:1789760713.689191:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702869235532 time 702959273164 state 0x2 ref 0 00000001:00080000:2.0:1789760713.689413:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971998 is committed 00000100:00100000:1.0:1789760713.693229:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760713.693855:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484160768 00000100:00100000:2.0:1789760713.693870:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484160768 00000100:00100000:2.0:1789760713.693946:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702869235532 time 702959273164 state 0x0 ref 1 00000100:00100000:2.0:1789760713.693964:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702959533580 time 702959273164 state 0x1 ref 1 00000100:00100000:2.0:1789760713.694001:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702959533580 time 703024930741 state 0x0 ref 1 00000100:00100000:2.0:1789760713.694016:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484160768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760713.694114:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19173376, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:2.0:1789760713.694125:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.694144:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:2.0:1789760713.694149:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760713.694157:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19173376->19177471] (req 19173376->19177471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760713.694173:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19173376->19177471] (req 19173376->19177471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760713.784964:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294971999 is committed 00002000:00010000:1.0:1789760713.791230:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19173376 -> 19177472 00010000:00010000:1.0:1789760713.791259:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19173376->19177471] (req 19173376->19177471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.791275:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19173376->19177471] (req 19173376->19177471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.791291:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19173376->19177471] (req 19173376->19177471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.791300:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760713.791305:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19173376->19177471] (req 19173376->19177471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.791316:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19173376->19177471] (req 19173376->19177471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.791341:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19173376->19177471] (req 19173376->19177471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.791350:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf38263962079bc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19173376->19177471] (req 19173376->19177471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760713.791489:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484160768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 97469us (98268us total) trans 4294971999 rc 0/0 00000100:00100000:1.0:1789760713.791501:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4710 00000100:00100000:1.0:1789760713.791514:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702959533580 time 703024930741 state 0x2 ref 0 00000100:00100000:0.0:1789760713.797616:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760713.797882:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484160896 00000100:00100000:3.0:1789760713.797896:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484160896 00000100:00100000:3.0:1789760713.797979:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 702959533580 time 703024930741 state 0x0 ref 1 00000100:00100000:3.0:1789760713.797999:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703025191157 time 703024930741 state 0x1 ref 1 00000100:00100000:3.0:1789760713.798033:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703025191157 time 703128964347 state 0x0 ref 1 00000100:00100000:3.0:1789760713.798048:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484160896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760713.798146:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19177472, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:3.0:1789760713.798156:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.798175:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:3.0:1789760713.798180:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760713.798187:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19177472->19181567] (req 19177472->19181567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.798204:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19177472->19181567] (req 19177472->19181567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760713.901057:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19177472 -> 19181568 00010000:00010000:3.0:1789760713.901111:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19177472->19181567] (req 19177472->19181567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.901128:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19177472->19181567] (req 19177472->19181567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.901143:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19177472->19181567] (req 19177472->19181567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.901151:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760713.901157:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19177472->19181567] (req 19177472->19181567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.901167:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19177472->19181567] (req 19177472->19181567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.901194:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19177472->19181567] (req 19177472->19181567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.901203:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079c3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19177472->19181567] (req 19177472->19181567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760713.901382:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484160896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 103328us (103773us total) trans 4294972000 rc 0/0 00000100:00100000:2.0:1789760713.901398:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4711 00000100:00100000:2.0:1789760713.901409:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703025191157 time 703128964347 state 0x2 ref 0 00000001:00080000:1.0:1789760713.901499:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972000 is committed 00000100:00100000:2.0:1789760713.903856:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760713.903999:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484161024 00000100:00100000:3.0:1789760713.904012:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484161024 00000100:00100000:3.0:1789760713.904118:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703025191157 time 703128964347 state 0x0 ref 1 00000100:00100000:3.0:1789760713.904138:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703129224763 time 703128964347 state 0x1 ref 1 00000100:00100000:3.0:1789760713.904174:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703129224763 time 703235104549 state 0x0 ref 1 00000100:00100000:3.0:1789760713.904189:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484161024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760713.904255:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19181568, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:3.0:1789760713.904266:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.904287:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:3.0:1789760713.904292:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760713.904299:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19181568->19185663] (req 19181568->19185663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760713.904318:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19181568->19185663] (req 19181568->19185663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760713.969823:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972001 is committed 00002000:00010000:1.0:1789760713.969994:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19181568 -> 19185664 00010000:00010000:1.0:1789760713.970010:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19181568->19185663] (req 19181568->19185663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.970029:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19181568->19185663] (req 19181568->19185663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.970038:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19181568->19185663] (req 19181568->19185663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.970045:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760713.970050:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19181568->19185663] (req 19181568->19185663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.970059:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19181568->19185663] (req 19181568->19185663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.970099:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19181568->19185663] (req 19181568->19185663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.970105:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962079ca lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19181568->19185663] (req 19181568->19185663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760713.970230:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484161024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66040us (66382us total) trans 4294972001 rc 0/0 00000100:00100000:1.0:1789760713.970240:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4712 00000100:00100000:1.0:1789760713.970249:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703129224763 time 703235104549 state 0x2 ref 0 00000100:00100000:0.0:1789760713.977254:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760713.977354:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484161152 00000100:00100000:1.0:1789760713.977364:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484161152 00000100:00100000:1.0:1789760713.977417:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703129224763 time 703235104549 state 0x0 ref 1 00000100:00100000:1.0:1789760713.977434:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703235364965 time 703235104549 state 0x1 ref 1 00000100:00100000:1.0:1789760713.977515:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703235364965 time 703308445371 state 0x0 ref 1 00000100:00100000:1.0:1789760713.977549:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484161152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760713.977610:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19185664, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:1.0:1789760713.977619:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.977635:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:1.0:1789760713.977640:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760713.977646:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19185664->19189759] (req 19185664->19189759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760713.977661:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19185664->19189759] (req 19185664->19189759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760714.014795:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972002 is committed 00002000:00010000:1.0:1789760714.014960:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19185664 -> 19189760 00010000:00010000:1.0:1789760714.014975:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19185664->19189759] (req 19185664->19189759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.014988:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19185664->19189759] (req 19185664->19189759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.015003:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19185664->19189759] (req 19185664->19189759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.015010:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760714.015015:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19185664->19189759] (req 19185664->19189759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.015025:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19185664->19189759] (req 19185664->19189759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.015043:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19185664->19189759] (req 19185664->19189759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.015051:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19185664->19189759] (req 19185664->19189759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760714.015164:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484161152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37622us (37916us total) trans 4294972002 rc 0/0 00000100:00100000:1.0:1789760714.015175:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4713 00000100:00100000:1.0:1789760714.015184:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703235364965 time 703308445371 state 0x2 ref 0 00000100:00100000:0.0:1789760714.020907:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760714.021090:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484161280 00000100:00100000:1.0:1789760714.021100:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484161280 00000100:00100000:1.0:1789760714.021184:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703235364965 time 703308445371 state 0x0 ref 1 00000100:00100000:1.0:1789760714.021199:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703308705787 time 703308445371 state 0x1 ref 1 00000100:00100000:1.0:1789760714.021233:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703308705787 time 703352163855 state 0x0 ref 1 00000100:00100000:1.0:1789760714.021245:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484161280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760714.021290:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19189760, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:1.0:1789760714.021299:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.021328:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:1.0:1789760714.021333:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760714.021340:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19189760->19193855] (req 19189760->19193855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.021354:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19189760->19193855] (req 19189760->19193855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760714.023815:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:3.0:1789760714.081688:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972003 is committed 00002000:00010000:1.0:1789760714.081872:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19189760 -> 19193856 00010000:00010000:1.0:1789760714.081889:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19189760->19193855] (req 19189760->19193855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.081901:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19189760->19193855] (req 19189760->19193855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.081909:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19189760->19193855] (req 19189760->19193855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.081916:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760714.081921:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19189760->19193855] (req 19189760->19193855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.081930:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19189760->19193855] (req 19189760->19193855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.081948:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19189760->19193855] (req 19189760->19193855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.081954:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962079d8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19189760->19193855] (req 19189760->19193855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760714.082045:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484161280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60798us (61145us total) trans 4294972003 rc 0/0 00000100:00100000:1.0:1789760714.082055:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4714 00000100:00100000:1.0:1789760714.082102:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703308705787 time 703352163855 state 0x2 ref 0 00000100:00100000:3.0:1789760714.088778:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760714.088916:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484161408 00000100:00100000:2.0:1789760714.088927:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484161408 00000100:00100000:2.0:1789760714.088977:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703308705787 time 703352163855 state 0x0 ref 1 00000100:00100000:2.0:1789760714.088992:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703352424271 time 703352163855 state 0x1 ref 1 00000100:00100000:2.0:1789760714.089019:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703352424271 time 703419950533 state 0x0 ref 1 00000100:00100000:2.0:1789760714.089030:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484161408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760714.089104:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19193856, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:2.0:1789760714.089113:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.089128:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:2.0:1789760714.089133:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760714.089140:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19193856->19197951] (req 19193856->19197951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.089169:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19193856->19197951] (req 19193856->19197951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760714.204638:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972004 is committed 00002000:00010000:2.0:1789760714.205116:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19193856 -> 19197952 00010000:00010000:2.0:1789760714.205133:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19193856->19197951] (req 19193856->19197951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.205146:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19193856->19197951] (req 19193856->19197951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.205157:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19193856->19197951] (req 19193856->19197951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.205163:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760714.205168:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19193856->19197951] (req 19193856->19197951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.205178:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19193856->19197951] (req 19193856->19197951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.205196:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19193856->19197951] (req 19193856->19197951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.205204:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962079df lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19193856->19197951] (req 19193856->19197951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760714.205294:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484161408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 116260us (116522us total) trans 4294972004 rc 0/0 00000100:00100000:2.0:1789760714.205315:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4715 00000100:00100000:2.0:1789760714.205325:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703352424271 time 703419950533 state 0x2 ref 0 00000100:00100000:2.0:1789760714.208702:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760714.208913:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484161536 00000100:00100000:3.0:1789760714.208923:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484161536 00000100:00100000:3.0:1789760714.208975:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703352424271 time 703419950533 state 0x0 ref 1 00000100:00100000:3.0:1789760714.209013:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703420210949 time 703419950533 state 0x1 ref 1 00000100:00100000:3.0:1789760714.209042:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703420210949 time 703539973013 state 0x0 ref 1 00000100:00100000:3.0:1789760714.209053:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484161536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760714.209135:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19197952, mtime: 0x6aad94c9, atime: 0x0, ctime: 0x6aad94c9, blocks: 0x9807 00010000:00010000:3.0:1789760714.209146:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.209163:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:3.0:1789760714.209167:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760714.209174:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19197952->19202047] (req 19197952->19202047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.209188:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19197952->19202047] (req 19197952->19202047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760714.271181:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972005 is committed 00002000:00010000:2.0:1789760714.271311:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19197952 -> 19202048 00002000:00010000:2.0:1789760714.271316:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760713 -> 1789760714 00002000:00010000:2.0:1789760714.271319:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760713 -> 1789760714 00010000:00010000:2.0:1789760714.271333:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19197952->19202047] (req 19197952->19202047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.271346:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19197952->19202047] (req 19197952->19202047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.271355:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19197952->19202047] (req 19197952->19202047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.271363:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760714.271368:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19197952->19202047] (req 19197952->19202047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.271377:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19197952->19202047] (req 19197952->19202047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.271402:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19197952->19202047] (req 19197952->19202047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.271410:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962079e6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19197952->19202047] (req 19197952->19202047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760714.271505:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484161536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62448us (62809us total) trans 4294972005 rc 0/0 00000100:00100000:2.0:1789760714.271516:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4716 00000100:00100000:2.0:1789760714.271537:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703420210949 time 703539973013 state 0x2 ref 0 00000100:00100000:1.0:1789760714.274667:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760714.274808:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484161664 00000100:00100000:0.0:1789760714.274828:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484161664 00000100:00100000:0.0:1789760714.274897:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703420210949 time 703539973013 state 0x0 ref 1 00000100:00100000:0.0:1789760714.274913:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703540233429 time 703539973013 state 0x1 ref 1 00000100:00100000:0.0:1789760714.274944:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703540233429 time 703605875533 state 0x0 ref 1 00000100:00100000:0.0:1789760714.274957:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484161664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760714.275003:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19202048, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:0.0:1789760714.275013:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.275034:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:0.0:1789760714.275038:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760714.275044:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19202048->19206143] (req 19202048->19206143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.275058:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19202048->19206143] (req 19202048->19206143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760714.351027:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19202048 -> 19206144 00010000:00010000:0.0:1789760714.351046:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19202048->19206143] (req 19202048->19206143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.351059:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19202048->19206143] (req 19202048->19206143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.351099:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19202048->19206143] (req 19202048->19206143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.351107:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760714.351112:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19202048->19206143] (req 19202048->19206143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.351121:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19202048->19206143] (req 19202048->19206143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.351139:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19202048->19206143] (req 19202048->19206143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.351146:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079ed lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19202048->19206143] (req 19202048->19206143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760714.351230:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484161664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 76271us (76569us total) trans 4294972006 rc 0/0 00000100:00100000:0.0:1789760714.351239:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4717 00000100:00100000:0.0:1789760714.351249:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703540233429 time 703605875533 state 0x2 ref 0 00000100:00100000:3.0:1789760714.354374:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760714.354796:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484161792 00000100:00100000:2.0:1789760714.354806:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484161792 00000100:00100000:2.0:1789760714.354858:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703540233429 time 703605875533 state 0x0 ref 1 00000100:00100000:2.0:1789760714.354874:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703606135949 time 703605875533 state 0x1 ref 1 00000100:00100000:2.0:1789760714.354902:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703606135949 time 703685833544 state 0x0 ref 1 00000100:00100000:2.0:1789760714.354915:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484161792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760714.354959:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19206144, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:2.0:1789760714.354969:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.354984:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:2.0:1789760714.354989:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760714.354995:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19206144->19210239] (req 19206144->19210239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.355009:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19206144->19210239] (req 19206144->19210239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760714.355810:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972006 is committed 00002000:00010000:1.0:1789760714.394876:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19206144 -> 19210240 00010000:00010000:1.0:1789760714.394897:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19206144->19210239] (req 19206144->19210239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.394910:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19206144->19210239] (req 19206144->19210239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.394918:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19206144->19210239] (req 19206144->19210239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.394926:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760714.394930:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19206144->19210239] (req 19206144->19210239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.394939:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19206144->19210239] (req 19206144->19210239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.394958:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19206144->19210239] (req 19206144->19210239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.394965:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962079f4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19206144->19210239] (req 19206144->19210239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760714.395131:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484161792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40213us (40762us total) trans 4294972007 rc 0/0 00000100:00100000:1.0:1789760714.395141:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4718 00000100:00100000:1.0:1789760714.395151:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703606135949 time 703685833544 state 0x2 ref 0 00000001:00080000:3.0:1789760714.395290:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972007 is committed 00000100:00100000:2.0:1789760714.399462:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760714.399564:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484161920 00000100:00100000:0.0:1789760714.399574:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484161920 00000100:00100000:0.0:1789760714.399632:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703606135949 time 703685833544 state 0x0 ref 1 00000100:00100000:0.0:1789760714.399648:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703686093960 time 703685833544 state 0x1 ref 1 00000100:00100000:0.0:1789760714.399680:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703686093960 time 703730611140 state 0x0 ref 1 00000100:00100000:0.0:1789760714.399694:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484161920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760714.399739:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19210240, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:0.0:1789760714.399747:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.399762:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:0.0:1789760714.399767:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760714.399773:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19210240->19214335] (req 19210240->19214335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760714.399792:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19210240->19214335] (req 19210240->19214335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760714.451547:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972008 is committed 00002000:00010000:2.0:1789760714.451679:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19210240 -> 19214336 00010000:00010000:2.0:1789760714.451696:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19210240->19214335] (req 19210240->19214335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.451709:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19210240->19214335] (req 19210240->19214335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.451719:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19210240->19214335] (req 19210240->19214335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.451726:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760714.451731:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19210240->19214335] (req 19210240->19214335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.451741:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19210240->19214335] (req 19210240->19214335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.451760:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19210240->19214335] (req 19210240->19214335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.451773:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962079fb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19210240->19214335] (req 19210240->19214335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760714.451888:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484161920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52191us (52430us total) trans 4294972008 rc 0/0 00000100:00100000:2.0:1789760714.451898:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4719 00000100:00100000:2.0:1789760714.451908:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703686093960 time 703730611140 state 0x2 ref 0 00000100:00100000:3.0:1789760714.462514:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760714.462627:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484162048 00000100:00100000:2.0:1789760714.462637:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484162048 00000100:00100000:2.0:1789760714.462691:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703686093960 time 703730611140 state 0x0 ref 1 00000100:00100000:2.0:1789760714.462707:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703730871556 time 703730611140 state 0x1 ref 1 00000100:00100000:2.0:1789760714.462740:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703730871556 time 703793671554 state 0x0 ref 1 00000100:00100000:2.0:1789760714.462754:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484162048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760714.462805:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19214336, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:2.0:1789760714.462815:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.462832:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:2.0:1789760714.462836:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760714.462843:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19214336->19218431] (req 19214336->19218431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.462872:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19214336->19218431] (req 19214336->19218431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760714.493944:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972009 is committed 00002000:00010000:3.0:1789760714.494935:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19214336 -> 19218432 00010000:00010000:3.0:1789760714.494952:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19214336->19218431] (req 19214336->19218431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.494965:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19214336->19218431] (req 19214336->19218431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.494974:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19214336->19218431] (req 19214336->19218431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.494981:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760714.494986:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19214336->19218431] (req 19214336->19218431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.494999:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19214336->19218431] (req 19214336->19218431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.495017:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19214336->19218431] (req 19214336->19218431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.495024:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207a02 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19214336->19218431] (req 19214336->19218431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760714.495284:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484162048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32527us (32773us total) trans 4294972009 rc 0/0 00000100:00100000:3.0:1789760714.495294:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4720 00000100:00100000:3.0:1789760714.495313:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703730871556 time 703793671554 state 0x2 ref 0 00000100:00100000:3.0:1789760714.518447:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760714.518511:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484162176 00000100:00100000:1.0:1789760714.518535:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484162176 00000100:00100000:1.0:1789760714.518860:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703730871556 time 703793671554 state 0x0 ref 1 00000100:00100000:1.0:1789760714.518877:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703793931970 time 703793671554 state 0x1 ref 1 00000100:00100000:1.0:1789760714.518917:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703793931970 time 703849848084 state 0x0 ref 1 00000100:00100000:1.0:1789760714.518931:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484162176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760714.519009:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19218432, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:1.0:1789760714.519019:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.519036:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:1.0:1789760714.519040:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760714.519046:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19218432->19222527] (req 19218432->19222527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.519059:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19218432->19222527] (req 19218432->19222527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760714.561331:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19218432 -> 19222528 00010000:00010000:2.0:1789760714.561352:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19218432->19222527] (req 19218432->19222527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.561364:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19218432->19222527] (req 19218432->19222527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.561374:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19218432->19222527] (req 19218432->19222527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.561381:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760714.561387:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19218432->19222527] (req 19218432->19222527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.561396:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19218432->19222527] (req 19218432->19222527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.561416:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19218432->19222527] (req 19218432->19222527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.561424:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a09 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19218432->19222527] (req 19218432->19222527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760714.561529:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484162176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42588us (43077us total) trans 4294972010 rc 0/0 00000100:00100000:2.0:1789760714.561539:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4721 00000100:00100000:2.0:1789760714.561549:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703793931970 time 703849848084 state 0x2 ref 0 00000001:00080000:3.0:1789760714.561787:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972010 is committed 00000100:00100000:2.0:1789760714.565966:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760714.566046:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484162304 00000100:00100000:1.0:1789760714.566056:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484162304 00000100:00100000:1.0:1789760714.566140:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703793931970 time 703849848084 state 0x0 ref 1 00000100:00100000:1.0:1789760714.566156:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703850108500 time 703849848084 state 0x1 ref 1 00000100:00100000:1.0:1789760714.566186:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703850108500 time 703897117327 state 0x0 ref 1 00000100:00100000:1.0:1789760714.566199:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484162304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760714.566248:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19222528, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:1.0:1789760714.566258:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.566274:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:1.0:1789760714.566278:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760714.566285:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19222528->19226623] (req 19222528->19226623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.566298:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19222528->19226623] (req 19222528->19226623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760714.611491:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972011 is committed 00002000:00010000:2.0:1789760714.612423:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19222528 -> 19226624 00010000:00010000:2.0:1789760714.612438:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19222528->19226623] (req 19222528->19226623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.612448:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19222528->19226623] (req 19222528->19226623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.612455:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19222528->19226623] (req 19222528->19226623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.612459:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760714.612463:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19222528->19226623] (req 19222528->19226623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.612469:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19222528->19226623] (req 19222528->19226623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.612491:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19222528->19226623] (req 19222528->19226623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.612497:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a10 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19222528->19226623] (req 19222528->19226623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760714.612668:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484162304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46466us (46706us total) trans 4294972011 rc 0/0 00000100:00100000:2.0:1789760714.612678:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4722 00000100:00100000:2.0:1789760714.612688:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703850108500 time 703897117327 state 0x2 ref 0 00000100:00100000:1.0:1789760714.616182:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760714.616292:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484162432 00000100:00100000:2.0:1789760714.616302:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484162432 00000100:00100000:2.0:1789760714.616354:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703850108500 time 703897117327 state 0x0 ref 1 00000100:00100000:2.0:1789760714.616370:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703897377743 time 703897117327 state 0x1 ref 1 00000100:00100000:2.0:1789760714.616403:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703897377743 time 703947333573 state 0x0 ref 1 00000100:00100000:2.0:1789760714.616416:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484162432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760714.616464:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19226624, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:2.0:1789760714.616472:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.616499:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:2.0:1789760714.616504:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760714.616510:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19226624->19230719] (req 19226624->19230719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.616540:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19226624->19230719] (req 19226624->19230719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.636228:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760714.653045:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972012 is committed 00002000:00010000:1.0:1789760714.653226:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19226624 -> 19230720 00010000:00010000:1.0:1789760714.653242:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19226624->19230719] (req 19226624->19230719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.653255:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19226624->19230719] (req 19226624->19230719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.653264:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19226624->19230719] (req 19226624->19230719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.653272:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760714.653277:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19226624->19230719] (req 19226624->19230719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.653286:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19226624->19230719] (req 19226624->19230719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.653318:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19226624->19230719] (req 19226624->19230719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.653327:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207a17 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19226624->19230719] (req 19226624->19230719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760714.653420:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484162432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37001us (37242us total) trans 4294972012 rc 0/0 00000100:00100000:1.0:1789760714.653431:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4723 00000100:00100000:1.0:1789760714.653441:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703897377743 time 703947333573 state 0x2 ref 0 00000100:00100000:1.0:1789760714.657206:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760714.657284:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484162560 00000100:00100000:3.0:1789760714.657293:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484162560 00000100:00100000:3.0:1789760714.657398:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703897377743 time 703947333573 state 0x0 ref 1 00000100:00100000:3.0:1789760714.657412:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703947593989 time 703947333573 state 0x1 ref 1 00000100:00100000:3.0:1789760714.657476:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703947593989 time 703988407135 state 0x0 ref 1 00000100:00100000:3.0:1789760714.657488:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484162560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760714.657543:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19230720, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:3.0:1789760714.657552:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.657567:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=703s v=(0 0) 00010000:00010000:3.0:1789760714.657571:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760714.657577:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19230720->19234815] (req 19230720->19234815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.657592:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19230720->19234815] (req 19230720->19234815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760714.695471:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972013 is committed 00002000:00010000:2.0:1789760714.695590:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19230720 -> 19234816 00010000:00010000:2.0:1789760714.695671:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19230720->19234815] (req 19230720->19234815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.695685:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19230720->19234815] (req 19230720->19234815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.695694:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19230720->19234815] (req 19230720->19234815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.695701:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760714.695706:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19230720->19234815] (req 19230720->19234815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.695715:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19230720->19234815] (req 19230720->19234815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.695737:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19230720->19234815] (req 19230720->19234815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.695744:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207a1e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19230720->19234815] (req 19230720->19234815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760714.695880:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484162560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38387us (38676us total) trans 4294972013 rc 0/0 00000100:00100000:2.0:1789760714.695890:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4724 00000100:00100000:2.0:1789760714.695899:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703947593989 time 703988407135 state 0x2 ref 0 00000100:00100000:2.0:1789760714.700747:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760714.700834:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484162688 00000100:00100000:1.0:1789760714.700843:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484162688 00000100:00100000:1.0:1789760714.700895:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703947593989 time 703988407135 state 0x0 ref 1 00000100:00100000:1.0:1789760714.700909:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703988667551 time 703988407135 state 0x1 ref 1 00000100:00100000:1.0:1789760714.700969:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703988667551 time 704031900468 state 0x0 ref 1 00000100:00100000:1.0:1789760714.700981:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484162688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760714.701029:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19234816, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:1.0:1789760714.701037:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.701058:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:1.0:1789760714.701063:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760714.701090:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19234816->19238911] (req 19234816->19238911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.701105:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19234816->19238911] (req 19234816->19238911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760714.732450:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972014 is committed 00002000:00010000:1.0:1789760714.732525:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19234816 -> 19238912 00010000:00010000:1.0:1789760714.732540:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19234816->19238911] (req 19234816->19238911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.732553:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19234816->19238911] (req 19234816->19238911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.732562:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19234816->19238911] (req 19234816->19238911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.732568:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760714.732573:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19234816->19238911] (req 19234816->19238911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.732582:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19234816->19238911] (req 19234816->19238911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.732663:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19234816->19238911] (req 19234816->19238911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.732674:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207a25 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19234816->19238911] (req 19234816->19238911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760714.732763:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484162688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31778us (32021us total) trans 4294972014 rc 0/0 00000100:00100000:1.0:1789760714.732772:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4725 00000100:00100000:1.0:1789760714.732781:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703988667551 time 704031900468 state 0x2 ref 0 00000100:00100000:1.0:1789760714.736221:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760714.736304:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484162816 00000100:00100000:2.0:1789760714.736315:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484162816 00000100:00100000:2.0:1789760714.736368:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 703988667551 time 704031900468 state 0x0 ref 1 00000100:00100000:2.0:1789760714.736382:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704032160884 time 704031900468 state 0x1 ref 1 00000100:00100000:2.0:1789760714.736412:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704032160884 time 704067343372 state 0x0 ref 1 00000100:00100000:2.0:1789760714.736426:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484162816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760714.736472:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19238912, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:2.0:1789760714.736481:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.736498:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:2.0:1789760714.736502:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760714.736508:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19238912->19243007] (req 19238912->19243007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.736531:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19238912->19243007] (req 19238912->19243007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760714.759969:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972015 is committed 00002000:00010000:3.0:1789760714.760235:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19238912 -> 19243008 00010000:00010000:3.0:1789760714.760250:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19238912->19243007] (req 19238912->19243007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.760263:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19238912->19243007] (req 19238912->19243007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.760273:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19238912->19243007] (req 19238912->19243007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.760281:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760714.760285:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19238912->19243007] (req 19238912->19243007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.760293:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19238912->19243007] (req 19238912->19243007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.760321:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19238912->19243007] (req 19238912->19243007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.760330:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207a2c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19238912->19243007] (req 19238912->19243007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760714.760411:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484162816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23983us (24195us total) trans 4294972015 rc 0/0 00000100:00100000:3.0:1789760714.760421:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4726 00000100:00100000:3.0:1789760714.760430:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704032160884 time 704067343372 state 0x2 ref 0 00000100:00100000:1.0:1789760714.763709:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760714.763810:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484162944 00000100:00100000:2.0:1789760714.763820:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484162944 00000100:00100000:2.0:1789760714.763876:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704032160884 time 704067343372 state 0x0 ref 1 00000100:00100000:2.0:1789760714.763891:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704067603788 time 704067343372 state 0x1 ref 1 00000100:00100000:2.0:1789760714.763953:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704067603788 time 704094883119 state 0x0 ref 1 00000100:00100000:2.0:1789760714.763966:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484162944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760714.764013:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19243008, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:2.0:1789760714.764023:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.764039:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:2.0:1789760714.764044:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760714.764050:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19243008->19247103] (req 19243008->19247103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.764393:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19243008->19247103] (req 19243008->19247103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760714.814339:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972016 is committed 00002000:00010000:1.0:1789760714.814451:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19243008 -> 19247104 00010000:00010000:1.0:1789760714.814467:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19243008->19247103] (req 19243008->19247103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.814480:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19243008->19247103] (req 19243008->19247103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.814489:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19243008->19247103] (req 19243008->19247103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.814497:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760714.814501:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19243008->19247103] (req 19243008->19247103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.814510:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19243008->19247103] (req 19243008->19247103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.814542:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19243008->19247103] (req 19243008->19247103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.814549:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207a33 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19243008->19247103] (req 19243008->19247103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760714.814727:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484162944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50757us (51020us total) trans 4294972016 rc 0/0 00000100:00100000:1.0:1789760714.814737:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4727 00000100:00100000:1.0:1789760714.814747:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704067603788 time 704094883119 state 0x2 ref 0 00000100:00100000:2.0:1789760714.818549:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760714.818783:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484163072 00000100:00100000:3.0:1789760714.818793:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484163072 00000100:00100000:3.0:1789760714.818850:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704067603788 time 704094883119 state 0x0 ref 1 00000100:00100000:3.0:1789760714.818867:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704095143535 time 704094883119 state 0x1 ref 1 00000100:00100000:3.0:1789760714.818896:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704095143535 time 704149827202 state 0x0 ref 1 00000100:00100000:3.0:1789760714.818909:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484163072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760714.818957:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19247104, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:3.0:1789760714.818965:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.818983:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:3.0:1789760714.818989:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760714.818995:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19247104->19251199] (req 19247104->19251199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.819009:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19247104->19251199] (req 19247104->19251199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760714.861407:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972017 is committed 00002000:00010000:3.0:1789760714.861461:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19247104 -> 19251200 00010000:00010000:3.0:1789760714.861476:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19247104->19251199] (req 19247104->19251199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.861485:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19247104->19251199] (req 19247104->19251199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.861491:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19247104->19251199] (req 19247104->19251199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.861495:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760714.861499:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19247104->19251199] (req 19247104->19251199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.861505:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19247104->19251199] (req 19247104->19251199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.861533:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19247104->19251199] (req 19247104->19251199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.861538:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a3a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19247104->19251199] (req 19247104->19251199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760714.861676:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484163072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42764us (43131us total) trans 4294972017 rc 0/0 00000100:00100000:3.0:1789760714.861686:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4728 00000100:00100000:3.0:1789760714.861695:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704095143535 time 704149827202 state 0x2 ref 0 00000100:00100000:0.0:1789760714.864368:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760714.864423:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484163200 00000100:00100000:1.0:1789760714.864433:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484163200 00000100:00100000:1.0:1789760714.864483:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704095143535 time 704149827202 state 0x0 ref 1 00000100:00100000:1.0:1789760714.864498:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704150087618 time 704149827202 state 0x1 ref 1 00000100:00100000:1.0:1789760714.864536:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704150087618 time 704195466453 state 0x0 ref 1 00000100:00100000:1.0:1789760714.864548:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484163200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760714.864594:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19251200, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:1.0:1789760714.864602:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.864622:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:1.0:1789760714.864626:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760714.864633:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19251200->19255295] (req 19251200->19255295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.864645:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19251200->19255295] (req 19251200->19255295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760714.896456:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972018 is committed 00002000:00010000:3.0:1789760714.896561:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19251200 -> 19255296 00010000:00010000:3.0:1789760714.896577:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19251200->19255295] (req 19251200->19255295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.896591:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19251200->19255295] (req 19251200->19255295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.896599:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19251200->19255295] (req 19251200->19255295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.896699:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760714.896709:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19251200->19255295] (req 19251200->19255295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.896716:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19251200->19255295] (req 19251200->19255295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.896733:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19251200->19255295] (req 19251200->19255295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.896741:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207a41 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19251200->19255295] (req 19251200->19255295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760714.896970:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484163200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32417us (32600us total) trans 4294972018 rc 0/0 00000100:00100000:3.0:1789760714.896981:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4729 00000100:00100000:3.0:1789760714.896991:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704150087618 time 704195466453 state 0x2 ref 0 00000100:00100000:1.0:1789760714.901272:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760714.901506:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484163328 00000100:00100000:2.0:1789760714.901528:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484163328 00000100:00100000:2.0:1789760714.901597:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704150087618 time 704195466453 state 0x0 ref 1 00000100:00100000:2.0:1789760714.901619:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704195726869 time 704195466453 state 0x1 ref 1 00000100:00100000:2.0:1789760714.901648:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704195726869 time 704232579771 state 0x0 ref 1 00000100:00100000:2.0:1789760714.901663:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484163328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760714.901711:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19255296, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:2.0:1789760714.901721:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.901737:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:2.0:1789760714.901743:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760714.901749:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19255296->19259391] (req 19255296->19259391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760714.901762:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19255296->19259391] (req 19255296->19259391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760714.938585:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972019 is committed 00002000:00010000:3.0:1789760714.939143:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19255296 -> 19259392 00010000:00010000:3.0:1789760714.939162:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19255296->19259391] (req 19255296->19259391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.939177:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19255296->19259391] (req 19255296->19259391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.939187:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19255296->19259391] (req 19255296->19259391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.939195:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760714.939200:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19255296->19259391] (req 19255296->19259391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.939209:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19255296->19259391] (req 19255296->19259391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.939234:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19255296->19259391] (req 19255296->19259391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.939242:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207a48 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19255296->19259391] (req 19255296->19259391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760714.939464:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484163328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37799us (38195us total) trans 4294972019 rc 0/0 00000100:00100000:3.0:1789760714.939476:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4730 00000100:00100000:3.0:1789760714.939486:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704195726869 time 704232579771 state 0x2 ref 0 00000100:00100000:2.0:1789760714.943693:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760714.943800:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484163456 00000100:00100000:1.0:1789760714.943812:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484163456 00000100:00100000:1.0:1789760714.944430:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704195726869 time 704232579771 state 0x0 ref 1 00000100:00100000:1.0:1789760714.944445:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704232840187 time 704232579771 state 0x1 ref 1 00000100:00100000:1.0:1789760714.944476:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704232840187 time 704275406714 state 0x0 ref 1 00000100:00100000:1.0:1789760714.944489:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484163456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760714.944547:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19259392, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:1.0:1789760714.944556:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.944571:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:1.0:1789760714.944576:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760714.944583:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19259392->19263487] (req 19259392->19263487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760714.944596:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19259392->19263487] (req 19259392->19263487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760714.982546:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972020 is committed 00002000:00010000:3.0:1789760714.982727:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19259392 -> 19263488 00010000:00010000:3.0:1789760714.982745:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19259392->19263487] (req 19259392->19263487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.982758:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19259392->19263487] (req 19259392->19263487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.982768:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19259392->19263487] (req 19259392->19263487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.982776:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760714.982781:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19259392->19263487] (req 19259392->19263487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.982797:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19259392->19263487] (req 19259392->19263487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.982818:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19259392->19263487] (req 19259392->19263487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.982825:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207a4f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19259392->19263487] (req 19259392->19263487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760714.983538:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484163456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39044us (39851us total) trans 4294972020 rc 0/0 00000100:00100000:3.0:1789760714.983550:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4731 00000100:00100000:3.0:1789760714.983560:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704232840187 time 704275406714 state 0x2 ref 0 00000100:00100000:0.0:1789760714.990049:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760714.990257:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484163584 00000100:00100000:3.0:1789760714.990268:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484163584 00000100:00100000:3.0:1789760714.990330:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704232840187 time 704275406714 state 0x0 ref 1 00000100:00100000:3.0:1789760714.990345:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704275667130 time 704275406714 state 0x1 ref 1 00000100:00100000:3.0:1789760714.990418:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704275667130 time 704321348802 state 0x0 ref 1 00000100:00100000:3.0:1789760714.990431:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484163584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760714.990476:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19263488, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:3.0:1789760714.990485:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.990500:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:3.0:1789760714.990504:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760714.990510:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19263488->19267583] (req 19263488->19267583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760714.990534:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19263488->19267583] (req 19263488->19267583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760715.031224:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19263488 -> 19267584 00010000:00010000:3.0:1789760715.031244:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19263488->19267583] (req 19263488->19267583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.031258:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19263488->19267583] (req 19263488->19267583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.031266:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19263488->19267583] (req 19263488->19267583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.031273:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760715.031278:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19263488->19267583] (req 19263488->19267583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.031288:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19263488->19267583] (req 19263488->19267583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.031318:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19263488->19267583] (req 19263488->19267583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.031327:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a56 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19263488->19267583] (req 19263488->19267583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760715.031415:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484163584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40981us (41370us total) trans 4294972021 rc 0/0 00000100:00100000:3.0:1789760715.031426:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4732 00000100:00100000:3.0:1789760715.031435:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704275667130 time 704321348802 state 0x2 ref 0 00000001:00080000:0.0:1789760715.032972:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972021 is committed 00000100:00100000:2.0:1789760715.035197:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760715.035276:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484163712 00000100:00100000:3.0:1789760715.035287:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484163712 00000100:00100000:3.0:1789760715.035403:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704275667130 time 704321348802 state 0x0 ref 1 00000100:00100000:3.0:1789760715.035419:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704321609218 time 704321348802 state 0x1 ref 1 00000100:00100000:3.0:1789760715.035473:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704321609218 time 704366405158 state 0x0 ref 1 00000100:00100000:3.0:1789760715.035484:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484163712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760715.035540:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19267584, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:3.0:1789760715.035549:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.035565:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:3.0:1789760715.035569:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760715.035576:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19267584->19271679] (req 19267584->19271679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.035639:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19267584->19271679] (req 19267584->19271679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760715.079234:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972022 is committed 00002000:00010000:3.0:1789760715.079410:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19267584 -> 19271680 00010000:00010000:3.0:1789760715.079426:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19267584->19271679] (req 19267584->19271679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.079439:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19267584->19271679] (req 19267584->19271679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.079448:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19267584->19271679] (req 19267584->19271679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.079455:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760715.079460:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19267584->19271679] (req 19267584->19271679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.079468:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19267584->19271679] (req 19267584->19271679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.079490:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19267584->19271679] (req 19267584->19271679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.079497:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a5d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19267584->19271679] (req 19267584->19271679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760715.079780:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484163712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44291us (44586us total) trans 4294972022 rc 0/0 00000100:00100000:3.0:1789760715.079790:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4733 00000100:00100000:3.0:1789760715.079800:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704321609218 time 704366405158 state 0x2 ref 0 00000100:00100000:2.0:1789760715.087664:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760715.087912:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484163840 00000100:00100000:0.0:1789760715.087922:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484163840 00000100:00100000:0.0:1789760715.087976:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704321609218 time 704366405158 state 0x0 ref 1 00000100:00100000:0.0:1789760715.087991:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704366665574 time 704366405158 state 0x1 ref 1 00000100:00100000:0.0:1789760715.088061:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704366665574 time 704418990074 state 0x0 ref 1 00000100:00100000:0.0:1789760715.088092:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484163840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760715.088143:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19271680, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:0.0:1789760715.088152:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760715.088166:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:0.0:1789760715.088170:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760715.088177:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19271680->19275775] (req 19271680->19275775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760715.088201:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19271680->19275775] (req 19271680->19275775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760715.120168:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972023 is committed 00002000:00010000:2.0:1789760715.120267:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19271680 -> 19275776 00010000:00010000:2.0:1789760715.120281:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19271680->19275775] (req 19271680->19275775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.120301:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19271680->19275775] (req 19271680->19275775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.120311:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19271680->19275775] (req 19271680->19275775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.120319:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760715.120323:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19271680->19275775] (req 19271680->19275775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.120333:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19271680->19275775] (req 19271680->19275775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.120405:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19271680->19275775] (req 19271680->19275775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.120414:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207a64 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19271680->19275775] (req 19271680->19275775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760715.120510:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484163840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32417us (32850us total) trans 4294972023 rc 0/0 00000100:00100000:2.0:1789760715.120534:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4734 00000100:00100000:2.0:1789760715.120544:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704366665574 time 704418990074 state 0x2 ref 0 00000100:00100000:1.0:1789760715.129559:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760715.129679:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484163968 00000100:00100000:3.0:1789760715.129699:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484163968 00000100:00100000:3.0:1789760715.129751:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704366665574 time 704418990074 state 0x0 ref 1 00000100:00100000:3.0:1789760715.129766:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704419250490 time 704418990074 state 0x1 ref 1 00000100:00100000:3.0:1789760715.129837:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704419250490 time 704460767702 state 0x0 ref 1 00000100:00100000:3.0:1789760715.129851:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484163968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760715.129896:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19275776, mtime: 0x6aad94ca, atime: 0x0, ctime: 0x6aad94ca, blocks: 0x9807 00010000:00010000:3.0:1789760715.129906:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.129922:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:3.0:1789760715.129927:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760715.129933:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19275776->19279871] (req 19275776->19279871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.129946:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19275776->19279871] (req 19275776->19279871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760715.193053:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972024 is committed 00002000:00010000:3.0:1789760715.193384:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19275776 -> 19279872 00002000:00010000:3.0:1789760715.193390:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760714 -> 1789760715 00002000:00010000:3.0:1789760715.193392:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760714 -> 1789760715 00010000:00010000:3.0:1789760715.193619:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19275776->19279871] (req 19275776->19279871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.193635:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19275776->19279871] (req 19275776->19279871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.193643:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19275776->19279871] (req 19275776->19279871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.193651:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760715.193656:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19275776->19279871] (req 19275776->19279871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.193666:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19275776->19279871] (req 19275776->19279871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.193685:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19275776->19279871] (req 19275776->19279871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.193693:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207a6b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19275776->19279871] (req 19275776->19279871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760715.193795:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484163968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63943us (64241us total) trans 4294972024 rc 0/0 00000100:00100000:3.0:1789760715.193805:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4735 00000100:00100000:3.0:1789760715.193815:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704419250490 time 704460767702 state 0x2 ref 0 00000100:00100000:1.0:1789760715.196994:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760715.197523:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484164096 00000100:00100000:3.0:1789760715.197534:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484164096 00000100:00100000:3.0:1789760715.197586:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704419250490 time 704460767702 state 0x0 ref 1 00000100:00100000:3.0:1789760715.197601:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704461028118 time 704460767702 state 0x1 ref 1 00000100:00100000:3.0:1789760715.197630:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704461028118 time 704528561680 state 0x0 ref 1 00000100:00100000:3.0:1789760715.197642:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484164096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760715.197686:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19279872, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:3.0:1789760715.197694:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.197710:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:3.0:1789760715.197715:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760715.197721:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19279872->19283967] (req 19279872->19283967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.197735:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19279872->19283967] (req 19279872->19283967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760715.240291:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972025 is committed 00002000:00010000:3.0:1789760715.240437:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19279872 -> 19283968 00010000:00010000:3.0:1789760715.240453:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19279872->19283967] (req 19279872->19283967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.240467:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19279872->19283967] (req 19279872->19283967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.240475:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19279872->19283967] (req 19279872->19283967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.240482:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760715.240487:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19279872->19283967] (req 19279872->19283967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.240497:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19279872->19283967] (req 19279872->19283967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.240525:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19279872->19283967] (req 19279872->19283967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.240534:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a72 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19279872->19283967] (req 19279872->19283967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760715.240632:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484164096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42986us (43640us total) trans 4294972025 rc 0/0 00000100:00100000:3.0:1789760715.240646:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4736 00000100:00100000:3.0:1789760715.240656:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704461028118 time 704528561680 state 0x2 ref 0 00000100:00100000:0.0:1789760715.255241:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760715.255331:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484164224 00000100:00100000:1.0:1789760715.255342:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484164224 00000100:00100000:1.0:1789760715.255394:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704461028118 time 704528561680 state 0x0 ref 1 00000100:00100000:1.0:1789760715.255409:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704528822096 time 704528561680 state 0x1 ref 1 00000100:00100000:1.0:1789760715.255440:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704528822096 time 704586371612 state 0x0 ref 1 00000100:00100000:1.0:1789760715.255453:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484164224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760715.255682:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19283968, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:1.0:1789760715.255692:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.255708:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:1.0:1789760715.255713:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760715.255721:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19283968->19288063] (req 19283968->19288063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.255737:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19283968->19288063] (req 19283968->19288063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760715.304774:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972026 is committed 00002000:00010000:3.0:1789760715.306244:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19283968 -> 19288064 00010000:00010000:3.0:1789760715.306262:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19283968->19288063] (req 19283968->19288063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.306275:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19283968->19288063] (req 19283968->19288063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.306283:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19283968->19288063] (req 19283968->19288063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.306291:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760715.306296:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19283968->19288063] (req 19283968->19288063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.306304:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19283968->19288063] (req 19283968->19288063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.306324:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19283968->19288063] (req 19283968->19288063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.306331:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207a79 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19283968->19288063] (req 19283968->19288063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760715.306437:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484164224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50981us (51202us total) trans 4294972026 rc 0/0 00000100:00100000:3.0:1789760715.306448:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4737 00000100:00100000:3.0:1789760715.306457:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704528822096 time 704586371612 state 0x2 ref 0 00000100:00100000:1.0:1789760715.312610:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760715.312883:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484164352 00000100:00100000:2.0:1789760715.312894:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484164352 00000100:00100000:2.0:1789760715.312950:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704528822096 time 704586371612 state 0x0 ref 1 00000100:00100000:2.0:1789760715.312966:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704586632028 time 704586371612 state 0x1 ref 1 00000100:00100000:2.0:1789760715.313039:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704586632028 time 704643968723 state 0x0 ref 1 00000100:00100000:2.0:1789760715.313052:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484164352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760715.313128:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19288064, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:2.0:1789760715.313137:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.313153:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:2.0:1789760715.313157:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760715.313173:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19288064->19292159] (req 19288064->19292159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.313189:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19288064->19292159] (req 19288064->19292159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760715.349035:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972027 is committed 00002000:00010000:2.0:1789760715.349238:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19288064 -> 19292160 00010000:00010000:2.0:1789760715.349255:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19288064->19292159] (req 19288064->19292159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.349266:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19288064->19292159] (req 19288064->19292159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.349279:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19288064->19292159] (req 19288064->19292159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.349287:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760715.349402:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19288064->19292159] (req 19288064->19292159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.349413:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19288064->19292159] (req 19288064->19292159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.349432:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19288064->19292159] (req 19288064->19292159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.349440:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207a80 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19288064->19292159] (req 19288064->19292159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760715.349536:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484164352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36481us (36930us total) trans 4294972027 rc 0/0 00000100:00100000:2.0:1789760715.349547:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4738 00000100:00100000:2.0:1789760715.349556:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704586632028 time 704643968723 state 0x2 ref 0 00000100:00100000:1.0:1789760715.371275:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760715.372404:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484164480 00000100:00100000:3.0:1789760715.372418:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484164480 00000100:00100000:3.0:1789760715.372479:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704586632028 time 704643968723 state 0x0 ref 1 00000100:00100000:3.0:1789760715.372494:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704644229139 time 704643968723 state 0x1 ref 1 00000100:00100000:3.0:1789760715.372539:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704644229139 time 704703469947 state 0x0 ref 1 00000100:00100000:3.0:1789760715.372553:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484164480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760715.372603:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19292160, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:3.0:1789760715.372612:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.372629:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:3.0:1789760715.372634:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760715.372640:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19292160->19296255] (req 19292160->19296255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.372653:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19292160->19296255] (req 19292160->19296255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760715.437463:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972028 is committed 00002000:00010000:1.0:1789760715.437580:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19292160 -> 19296256 00010000:00010000:1.0:1789760715.437594:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19292160->19296255] (req 19292160->19296255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.437616:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19292160->19296255] (req 19292160->19296255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.437625:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19292160->19296255] (req 19292160->19296255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.437632:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760715.437638:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19292160->19296255] (req 19292160->19296255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.437647:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19292160->19296255] (req 19292160->19296255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.437668:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19292160->19296255] (req 19292160->19296255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.437677:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207a87 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19292160->19296255] (req 19292160->19296255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760715.437773:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484164480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65218us (66500us total) trans 4294972028 rc 0/0 00000100:00100000:1.0:1789760715.437791:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4739 00000100:00100000:1.0:1789760715.437801:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704644229139 time 704703469947 state 0x2 ref 0 00000100:00100000:2.0:1789760715.446181:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760715.446313:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484164608 00000100:00100000:3.0:1789760715.446323:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484164608 00000100:00100000:3.0:1789760715.446376:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704644229139 time 704703469947 state 0x0 ref 1 00000100:00100000:3.0:1789760715.446393:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704703730363 time 704703469947 state 0x1 ref 1 00000100:00100000:3.0:1789760715.446420:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704703730363 time 704777351423 state 0x0 ref 1 00000100:00100000:3.0:1789760715.446432:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484164608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760715.446479:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19296256, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:3.0:1789760715.446487:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.446501:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:3.0:1789760715.446506:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760715.446522:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19296256->19300351] (req 19296256->19300351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.446536:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19296256->19300351] (req 19296256->19300351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760715.539721:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972029 is committed 00002000:00010000:1.0:1789760715.539828:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19296256 -> 19300352 00010000:00010000:1.0:1789760715.539843:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19296256->19300351] (req 19296256->19300351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.539855:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19296256->19300351] (req 19296256->19300351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.539863:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19296256->19300351] (req 19296256->19300351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.539870:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760715.539875:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19296256->19300351] (req 19296256->19300351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.539883:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19296256->19300351] (req 19296256->19300351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.539909:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19296256->19300351] (req 19296256->19300351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.539917:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207a8e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19296256->19300351] (req 19296256->19300351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760715.540007:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484164608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 93572us (93831us total) trans 4294972029 rc 0/0 00000100:00100000:1.0:1789760715.540017:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4740 00000100:00100000:1.0:1789760715.540027:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704703730363 time 704777351423 state 0x2 ref 0 00000100:00100000:3.0:1789760715.543179:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760715.543244:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484164736 00000100:00100000:1.0:1789760715.543257:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484164736 00000100:00100000:1.0:1789760715.543310:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704703730363 time 704777351423 state 0x0 ref 1 00000100:00100000:1.0:1789760715.543322:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704777611839 time 704777351423 state 0x1 ref 1 00000100:00100000:1.0:1789760715.543351:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704777611839 time 704874283059 state 0x0 ref 1 00000100:00100000:1.0:1789760715.543360:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484164736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760715.543399:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19300352, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:1.0:1789760715.543405:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.543416:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:1.0:1789760715.543419:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760715.543423:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19300352->19304447] (req 19300352->19304447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.543433:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19300352->19304447] (req 19300352->19304447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760715.585891:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972030 is committed 00002000:00010000:2.0:1789760715.586677:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19300352 -> 19304448 00010000:00010000:2.0:1789760715.586694:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19300352->19304447] (req 19300352->19304447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.586712:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19300352->19304447] (req 19300352->19304447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.586721:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19300352->19304447] (req 19300352->19304447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.586728:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760715.586732:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19300352->19304447] (req 19300352->19304447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.586740:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19300352->19304447] (req 19300352->19304447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.586759:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19300352->19304447] (req 19300352->19304447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.586766:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207a95 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19300352->19304447] (req 19300352->19304447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760715.586857:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484164736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43493us (43705us total) trans 4294972030 rc 0/0 00000100:00100000:2.0:1789760715.586868:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4741 00000100:00100000:2.0:1789760715.586878:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704777611839 time 704874283059 state 0x2 ref 0 00000100:00100000:2.0:1789760715.590105:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760715.590231:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484164864 00000100:00100000:0.0:1789760715.590241:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484164864 00000100:00100000:0.0:1789760715.590305:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704777611839 time 704874283059 state 0x0 ref 1 00000100:00100000:0.0:1789760715.590321:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704874543475 time 704874283059 state 0x1 ref 1 00000100:00100000:0.0:1789760715.590350:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704874543475 time 704921281323 state 0x0 ref 1 00000100:00100000:0.0:1789760715.590363:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484164864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760715.590411:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19304448, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:0.0:1789760715.590419:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760715.590435:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:0.0:1789760715.590439:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760715.590446:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19304448->19308543] (req 19304448->19308543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760715.590460:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19304448->19308543] (req 19304448->19308543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760715.617571:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972031 is committed 00002000:00010000:1.0:1789760715.617673:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19304448 -> 19308544 00010000:00010000:1.0:1789760715.617690:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19304448->19308543] (req 19304448->19308543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.617703:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19304448->19308543] (req 19304448->19308543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.617712:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19304448->19308543] (req 19304448->19308543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.617720:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760715.617725:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19304448->19308543] (req 19304448->19308543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.617735:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19304448->19308543] (req 19304448->19308543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.617754:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19304448->19308543] (req 19304448->19308543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.617762:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207a9c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19304448->19308543] (req 19304448->19308543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760715.617850:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484164864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27485us (27748us total) trans 4294972031 rc 0/0 00000100:00100000:1.0:1789760715.617868:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4742 00000100:00100000:1.0:1789760715.617879:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704874543475 time 704921281323 state 0x2 ref 0 00000100:00100000:0.0:1789760715.629777:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760715.629875:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484164992 00000100:00100000:3.0:1789760715.629885:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484164992 00000100:00100000:3.0:1789760715.629937:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704874543475 time 704921281323 state 0x0 ref 1 00000100:00100000:3.0:1789760715.629952:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704921541739 time 704921281323 state 0x1 ref 1 00000100:00100000:3.0:1789760715.629986:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704921541739 time 704960917268 state 0x0 ref 1 00000100:00100000:3.0:1789760715.630000:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484164992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760715.630046:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19308544, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:3.0:1789760715.630055:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.630386:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=704s v=(0 0) 00010000:00010000:3.0:1789760715.630391:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760715.630398:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19308544->19312639] (req 19308544->19312639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.630412:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19308544->19312639] (req 19308544->19312639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.660763:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760715.666570:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972032 is committed 00002000:00010000:1.0:1789760715.666739:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19308544 -> 19312640 00010000:00010000:1.0:1789760715.666754:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19308544->19312639] (req 19308544->19312639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.666766:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19308544->19312639] (req 19308544->19312639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.666775:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19308544->19312639] (req 19308544->19312639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.666846:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760715.666851:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19308544->19312639] (req 19308544->19312639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.666860:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19308544->19312639] (req 19308544->19312639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.666985:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19308544->19312639] (req 19308544->19312639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.667000:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207aa3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19308544->19312639] (req 19308544->19312639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760715.667209:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484164992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37207us (37533us total) trans 4294972032 rc 0/0 00000100:00100000:1.0:1789760715.667219:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4743 00000100:00100000:1.0:1789760715.667230:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704921541739 time 704960917268 state 0x2 ref 0 00000100:00100000:2.0:1789760715.670658:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760715.670772:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484165120 00000100:00100000:1.0:1789760715.670791:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484165120 00000100:00100000:1.0:1789760715.670839:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704921541739 time 704960917268 state 0x0 ref 1 00000100:00100000:1.0:1789760715.670854:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704961177684 time 704960917268 state 0x1 ref 1 00000100:00100000:1.0:1789760715.670896:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704961177684 time 705001827382 state 0x0 ref 1 00000100:00100000:1.0:1789760715.670909:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484165120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760715.670955:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19312640, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:1.0:1789760715.670963:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.670978:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:1.0:1789760715.670983:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760715.670989:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19312640->19316735] (req 19312640->19316735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.671010:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19312640->19316735] (req 19312640->19316735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760715.729749:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972033 is committed 00002000:00010000:1.0:1789760715.729857:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19312640 -> 19316736 00010000:00010000:1.0:1789760715.729872:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19312640->19316735] (req 19312640->19316735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.729978:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19312640->19316735] (req 19312640->19316735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.729989:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19312640->19316735] (req 19312640->19316735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.729995:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760715.730001:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19312640->19316735] (req 19312640->19316735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.730008:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19312640->19316735] (req 19312640->19316735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.730027:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19312640->19316735] (req 19312640->19316735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760715.730034:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207aaa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19312640->19316735] (req 19312640->19316735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760715.730156:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484165120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59244us (59503us total) trans 4294972033 rc 0/0 00000100:00100000:1.0:1789760715.730166:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4744 00000100:00100000:1.0:1789760715.730175:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704961177684 time 705001827382 state 0x2 ref 0 00000100:00100000:0.0:1789760715.736019:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760715.736114:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484165248 00000100:00100000:2.0:1789760715.736124:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484165248 00000100:00100000:2.0:1789760715.736175:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 704961177684 time 705001827382 state 0x0 ref 1 00000100:00100000:2.0:1789760715.736191:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705002087798 time 705001827382 state 0x1 ref 1 00000100:00100000:2.0:1789760715.736274:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705002087798 time 705067204479 state 0x0 ref 1 00000100:00100000:2.0:1789760715.736287:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b18e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484165248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760715.736337:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19316736, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:2.0:1789760715.736347:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.736365:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:2.0:1789760715.736369:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760715.736376:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19316736->19320831] (req 19316736->19320831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.736391:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19316736->19320831] (req 19316736->19320831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760715.803648:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19316736 -> 19320832 00010000:00010000:3.0:1789760715.803666:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19316736->19320831] (req 19316736->19320831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.803679:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19316736->19320831] (req 19316736->19320831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.803690:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19316736->19320831] (req 19316736->19320831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.803697:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760715.803702:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19316736->19320831] (req 19316736->19320831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.803711:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19316736->19320831] (req 19316736->19320831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.803729:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19316736->19320831] (req 19316736->19320831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.803736:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207ab1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19316736->19320831] (req 19316736->19320831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760715.803833:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484165248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67542us (67818us total) trans 4294972034 rc 0/0 00000100:00100000:3.0:1789760715.803843:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4745 00000100:00100000:3.0:1789760715.803853:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705002087798 time 705067204479 state 0x2 ref 0 00000001:00080000:2.0:1789760715.804383:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972034 is committed 00000100:00100000:0.0:1789760715.816053:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760715.816164:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484165376 00000100:00100000:1.0:1789760715.816175:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484165376 00000100:00100000:2.0:1789760715.816293:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705002087798 time 705067204479 state 0x0 ref 1 00000100:00100000:2.0:1789760715.816306:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705067464895 time 705067204479 state 0x1 ref 1 00000100:00100000:2.0:1789760715.816331:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705067464895 time 705147263322 state 0x0 ref 1 00000100:00100000:2.0:1789760715.816340:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484165376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760715.816378:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19320832, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:2.0:1789760715.816385:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.816398:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:2.0:1789760715.816402:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760715.816406:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19320832->19324927] (req 19320832->19324927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.816417:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19320832->19324927] (req 19320832->19324927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760715.842094:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19320832 -> 19324928 00010000:00010000:2.0:1789760715.842115:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19320832->19324927] (req 19320832->19324927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.842129:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19320832->19324927] (req 19320832->19324927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.842139:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19320832->19324927] (req 19320832->19324927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.842146:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760715.842152:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19320832->19324927] (req 19320832->19324927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.842161:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19320832->19324927] (req 19320832->19324927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.842181:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19320832->19324927] (req 19320832->19324927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.842191:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ab8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19320832->19324927] (req 19320832->19324927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760715.842283:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484165376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25938us (26233us total) trans 4294972035 rc 0/0 00000100:00100000:2.0:1789760715.842303:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4746 00000100:00100000:2.0:1789760715.842312:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705067464895 time 705147263322 state 0x2 ref 0 00000001:00080000:3.0:1789760715.842444:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972035 is committed 00000100:00100000:1.0:1789760715.846590:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760715.846787:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484165504 00000100:00100000:2.0:1789760715.846797:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484165504 00000100:00100000:2.0:1789760715.846921:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705067464895 time 705147263322 state 0x0 ref 1 00000100:00100000:2.0:1789760715.846940:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705147523738 time 705147263322 state 0x1 ref 1 00000100:00100000:2.0:1789760715.847011:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705147523738 time 705177941428 state 0x0 ref 1 00000100:00100000:2.0:1789760715.847024:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484165504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760715.847113:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19324928, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:2.0:1789760715.847123:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.847139:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:2.0:1789760715.847143:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760715.847148:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19324928->19329023] (req 19324928->19329023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.847161:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19324928->19329023] (req 19324928->19329023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760715.871315:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972036 is committed 00002000:00010000:3.0:1789760715.871614:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19324928 -> 19329024 00010000:00010000:3.0:1789760715.871630:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19324928->19329023] (req 19324928->19329023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.871650:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19324928->19329023] (req 19324928->19329023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.871659:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19324928->19329023] (req 19324928->19329023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.871668:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760715.871673:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19324928->19329023] (req 19324928->19329023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.871682:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19324928->19329023] (req 19324928->19329023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.871702:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19324928->19329023] (req 19324928->19329023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.871709:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396207abf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19324928->19329023] (req 19324928->19329023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760715.871796:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484165504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24770us (25212us total) trans 4294972036 rc 0/0 00000100:00100000:3.0:1789760715.871806:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4747 00000100:00100000:3.0:1789760715.871815:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705147523738 time 705177941428 state 0x2 ref 0 00000100:00100000:1.0:1789760715.876522:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760715.876626:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484165632 00000100:00100000:2.0:1789760715.876635:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484165632 00000100:00100000:2.0:1789760715.876687:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705147523738 time 705177941428 state 0x0 ref 1 00000100:00100000:2.0:1789760715.876700:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705178201844 time 705177941428 state 0x1 ref 1 00000100:00100000:2.0:1789760715.876733:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705178201844 time 705207663612 state 0x0 ref 1 00000100:00100000:2.0:1789760715.876745:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484165632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760715.876802:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19329024, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:2.0:1789760715.876811:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.876827:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:2.0:1789760715.876832:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760715.876838:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19329024->19333119] (req 19329024->19333119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.876863:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19329024->19333119] (req 19329024->19333119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760715.915611:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972037 is committed 00002000:00010000:3.0:1789760715.915681:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19329024 -> 19333120 00010000:00010000:3.0:1789760715.915696:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19329024->19333119] (req 19329024->19333119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.915709:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19329024->19333119] (req 19329024->19333119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.915719:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19329024->19333119] (req 19329024->19333119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.915727:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760715.915732:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19329024->19333119] (req 19329024->19333119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.915742:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19329024->19333119] (req 19329024->19333119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.915763:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19329024->19333119] (req 19329024->19333119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.915772:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207ac6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19329024->19333119] (req 19329024->19333119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760715.916011:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484165632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39262us (39504us total) trans 4294972037 rc 0/0 00000100:00100000:3.0:1789760715.916123:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4748 00000100:00100000:3.0:1789760715.916133:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705178201844 time 705207663612 state 0x2 ref 0 00000100:00100000:0.0:1789760715.927584:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760715.928097:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484165760 00000100:00100000:3.0:1789760715.928107:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484165760 00000100:00100000:3.0:1789760715.928163:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705178201844 time 705207663612 state 0x0 ref 1 00000100:00100000:3.0:1789760715.928179:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705207924028 time 705207663612 state 0x1 ref 1 00000100:00100000:3.0:1789760715.928214:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705207924028 time 705259145129 state 0x0 ref 1 00000100:00100000:3.0:1789760715.928227:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484165760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760715.928277:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19333120, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:3.0:1789760715.928286:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.928459:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:3.0:1789760715.928464:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760715.928728:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19333120->19337215] (req 19333120->19337215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760715.928745:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19333120->19337215] (req 19333120->19337215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760715.950248:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972038 is committed 00002000:00010000:2.0:1789760715.950395:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19333120 -> 19337216 00010000:00010000:2.0:1789760715.950412:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19333120->19337215] (req 19333120->19337215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.950424:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19333120->19337215] (req 19333120->19337215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.950432:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19333120->19337215] (req 19333120->19337215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.950437:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760715.950441:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19333120->19337215] (req 19333120->19337215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.950446:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19333120->19337215] (req 19333120->19337215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.950462:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19333120->19337215] (req 19333120->19337215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.950467:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207acd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19333120->19337215] (req 19333120->19337215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760715.950624:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484165760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 22394us (23044us total) trans 4294972038 rc 0/0 00000100:00100000:2.0:1789760715.950634:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4749 00000100:00100000:2.0:1789760715.950643:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705207924028 time 705259145129 state 0x2 ref 0 00000100:00100000:1.0:1789760715.961028:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760715.962397:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484165888 00000100:00100000:0.0:1789760715.962409:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484165888 00000100:00100000:0.0:1789760715.962471:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705207924028 time 705259145129 state 0x0 ref 1 00000100:00100000:0.0:1789760715.962486:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705259405545 time 705259145129 state 0x1 ref 1 00000100:00100000:0.0:1789760715.962579:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705259405545 time 705293508983 state 0x0 ref 1 00000100:00100000:0.0:1789760715.962602:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484165888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760715.962649:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19337216, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:0.0:1789760715.962658:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760715.962674:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:0.0:1789760715.962678:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760715.962685:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19337216->19341311] (req 19337216->19341311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760715.962699:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19337216->19341311] (req 19337216->19341311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760715.983688:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972039 is committed 00002000:00010000:2.0:1789760715.984111:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19337216 -> 19341312 00010000:00010000:2.0:1789760715.984129:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19337216->19341311] (req 19337216->19341311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.984143:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19337216->19341311] (req 19337216->19341311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.984153:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19337216->19341311] (req 19337216->19341311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.984160:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760715.984165:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19337216->19341311] (req 19337216->19341311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.984171:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19337216->19341311] (req 19337216->19341311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.984187:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19337216->19341311] (req 19337216->19341311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.984195:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207ad4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19337216->19341311] (req 19337216->19341311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760715.984359:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484165888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 21763us (23334us total) trans 4294972039 rc 0/0 00000100:00100000:3.0:1789760715.984370:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4750 00000100:00100000:3.0:1789760715.984380:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705259405545 time 705293508983 state 0x2 ref 0 00000100:00100000:1.0:1789760715.987501:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760715.987640:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484166016 00000100:00100000:2.0:1789760715.987651:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484166016 00000100:00100000:2.0:1789760715.987708:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705259405545 time 705293508983 state 0x0 ref 1 00000100:00100000:2.0:1789760715.987724:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705293769399 time 705293508983 state 0x1 ref 1 00000100:00100000:2.0:1789760715.987757:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705293769399 time 705318687999 state 0x0 ref 1 00000100:00100000:2.0:1789760715.987772:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484166016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760715.987824:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19341312, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:2.0:1789760715.987834:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.987851:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:2.0:1789760715.987856:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760715.987863:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19341312->19345407] (req 19341312->19345407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760715.987888:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19341312->19345407] (req 19341312->19345407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760716.024021:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972040 is committed 00002000:00010000:0.0:1789760716.024147:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19341312 -> 19345408 00010000:00010000:0.0:1789760716.024166:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19341312->19345407] (req 19341312->19345407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.024179:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19341312->19345407] (req 19341312->19345407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.024188:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19341312->19345407] (req 19341312->19345407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.024195:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760716.024200:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19341312->19345407] (req 19341312->19345407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.024209:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19341312->19345407] (req 19341312->19345407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.024229:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19341312->19345407] (req 19341312->19345407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.024237:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207adb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19341312->19345407] (req 19341312->19345407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760716.024334:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484166016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36560us (36838us total) trans 4294972040 rc 0/0 00000100:00100000:0.0:1789760716.024349:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4751 00000100:00100000:0.0:1789760716.024360:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705293769399 time 705318687999 state 0x2 ref 0 00000100:00100000:0.0:1789760716.028628:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760716.028697:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484166144 00000100:00100000:3.0:1789760716.028709:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484166144 00000100:00100000:3.0:1789760716.028781:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705293769399 time 705318687999 state 0x0 ref 1 00000100:00100000:3.0:1789760716.028797:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705318948415 time 705318687999 state 0x1 ref 1 00000100:00080000:2.0:1789760716.028985:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760716.029025:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705318948415 time 705359759956 state 0x0 ref 1 00000100:00100000:3.0:1789760716.029041:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484166144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760716.029121:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19345408, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:3.0:1789760716.029130:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.029146:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:3.0:1789760716.029150:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760716.029157:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19345408->19349503] (req 19345408->19349503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.029171:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19345408->19349503] (req 19345408->19349503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760716.083025:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972041 is committed 00002000:00010000:0.0:1789760716.083850:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19345408 -> 19349504 00010000:00010000:0.0:1789760716.083876:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19345408->19349503] (req 19345408->19349503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.083889:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19345408->19349503] (req 19345408->19349503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.083898:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19345408->19349503] (req 19345408->19349503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.083905:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760716.083910:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19345408->19349503] (req 19345408->19349503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.083918:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19345408->19349503] (req 19345408->19349503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.083937:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19345408->19349503] (req 19345408->19349503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.083945:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207ae2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19345408->19349503] (req 19345408->19349503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760716.084048:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484166144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55006us (55426us total) trans 4294972041 rc 0/0 00000100:00100000:0.0:1789760716.084059:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4752 00000100:00100000:0.0:1789760716.084105:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705318948415 time 705359759956 state 0x2 ref 0 00000100:00100000:1.0:1789760716.088593:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760716.088684:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484166272 00000100:00100000:3.0:1789760716.088694:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484166272 00000100:00100000:3.0:1789760716.088752:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705318948415 time 705359759956 state 0x0 ref 1 00000100:00100000:3.0:1789760716.088768:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705360020372 time 705359759956 state 0x1 ref 1 00000100:00100000:3.0:1789760716.088799:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705360020372 time 705419730146 state 0x0 ref 1 00000100:00100000:3.0:1789760716.088812:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484166272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760716.088859:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19349504, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:3.0:1789760716.088869:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.088886:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:3.0:1789760716.088890:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760716.088897:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19349504->19353599] (req 19349504->19353599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.088911:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19349504->19353599] (req 19349504->19353599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760716.161649:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972042 is committed 00002000:00010000:3.0:1789760716.161846:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19349504 -> 19353600 00010000:00010000:3.0:1789760716.161867:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19349504->19353599] (req 19349504->19353599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.161880:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19349504->19353599] (req 19349504->19353599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.161888:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19349504->19353599] (req 19349504->19353599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.161896:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760716.161901:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19349504->19353599] (req 19349504->19353599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.161910:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19349504->19353599] (req 19349504->19353599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.161929:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19349504->19353599] (req 19349504->19353599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.161937:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207ae9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19349504->19353599] (req 19349504->19353599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760716.162029:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484166272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73215us (73440us total) trans 4294972042 rc 0/0 00000100:00100000:3.0:1789760716.162039:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4753 00000100:00100000:3.0:1789760716.162050:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705360020372 time 705419730146 state 0x2 ref 0 00000100:00100000:1.0:1789760716.168973:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760716.169135:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484166400 00000100:00100000:0.0:1789760716.169146:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484166400 00000100:00100000:0.0:1789760716.169213:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705360020372 time 705419730146 state 0x0 ref 1 00000100:00100000:0.0:1789760716.169230:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705419990562 time 705419730146 state 0x1 ref 1 00000100:00100000:0.0:1789760716.169267:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705419990562 time 705500197634 state 0x0 ref 1 00000100:00100000:0.0:1789760716.169281:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484166400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760716.169343:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19353600, mtime: 0x6aad94cb, atime: 0x0, ctime: 0x6aad94cb, blocks: 0x9807 00010000:00010000:0.0:1789760716.169352:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.169368:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:0.0:1789760716.169372:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760716.169378:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19353600->19357695] (req 19353600->19357695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.169393:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19353600->19357695] (req 19353600->19357695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760716.172703:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508908800:0@lo:13:osp-pre-1-0.0 00000100:00100000:3.0:1789760716.172773:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760716.172862:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508908800 00000100:00100000:3.0:1789760716.172869:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508908800 00000100:00100000:3.0:1789760716.172921:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 360 00000100:00100000:3.0:1789760716.172931:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508908800:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:3.0:1789760716.172964:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:3.0:1789760716.173006:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508908800:12345-0@lo:13:osp-pre-1-0.0 Request processed in 78us (233us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760716.173027:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 360 00000100:00080000:3.0:1789760716.173047:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:3.0:1789760716.173099:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508908800:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760716.173564:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c12680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508909056:0@lo:400:kworker.0 00000100:00100000:1.0:1789760716.173610:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760716.173632:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c12a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508909568:0@lo:400:kworker.0 00000100:00100000:1.0:1789760716.173651:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760716.173674:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c11180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508910080:0@lo:400:kworker.0 00000100:00100000:1.0:1789760716.173693:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760716.173712:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [3->2] 00000100:00100000:1.0:1789760716.173719:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c12300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508909184:0@lo:400:kworker.0 00000100:00100000:1.0:1789760716.173741:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760716.173760:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c10a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508909696:0@lo:400:kworker.0 00000100:00100000:1.0:1789760716.173778:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760716.173794:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c13b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508910208:0@lo:400:kworker.0 00000100:00100000:1.0:1789760716.173813:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760716.173842:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508909056 00000100:00100000:1.0:1789760716.173846:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508909056 00000100:00100000:1.0:1789760716.174215:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508909568 00000100:00100000:1.0:1789760716.174219:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508909568 00000100:00100000:1.0:1789760716.174236:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508910080 00000100:00100000:1.0:1789760716.174238:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508910080 00000100:00100000:1.0:1789760716.174248:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508909696 00000100:00100000:1.0:1789760716.174250:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508909696 00000100:00100000:1.0:1789760716.174260:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508910208 00000100:00100000:1.0:1789760716.174263:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508910208 00000100:00100000:1.0:1789760716.174276:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 865 00000100:00100000:1.0:1789760716.174282:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508909056:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.174745:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508909184 00000100:00100000:0.0:1789760716.174749:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508909184 00000100:00100000:0.0:1789760716.174783:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 419 00000100:00100000:0.0:1789760716.174791:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508909184:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.174865:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508909184:12345-0@lo:400:kworker.0 Request processed in 74us (1121us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760716.174872:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 419 00000100:00100000:0.0:1789760716.174896:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 866 00000100:00100000:0.0:1789760716.174904:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508909568:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.174932:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508909568:12345-0@lo:400:kworker.0 Request processed in 31us (1280us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760716.174939:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 866 00000100:00100000:0.0:1789760716.174955:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 867 00000100:00100000:0.0:1789760716.174962:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82dc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508910080:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.174989:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82dc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508910080:12345-0@lo:400:kworker.0 Request processed in 28us (1295us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760716.174995:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 867 00000100:00100000:0.0:1789760716.175006:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 868 00000100:00100000:0.0:1789760716.175024:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82f100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508909696:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.175048:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82f100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508909696:12345-0@lo:400:kworker.0 Request processed in 36us (1269us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760716.175053:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 868 00000100:00100000:0.0:1789760716.175083:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 869 00000100:00100000:0.0:1789760716.175090:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ed80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508910208:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.175117:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ed80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508910208:12345-0@lo:400:kworker.0 Request processed in 28us (1303us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760716.175123:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 869 00000100:00100000:0.0:1789760716.175145:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c11c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508909312:0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.175172:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760716.175197:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c13480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508909824:0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.175218:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760716.175234:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df82d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508910336:0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.175256:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760716.175274:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:0.0:1789760716.175280:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c11500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508908928:0@lo:400:kworker.0 00000100:00100000:1.0:1789760716.175468:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508909056:12345-0@lo:400:kworker.0 Request processed in 1186us (1857us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760716.175474:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 865 00000100:00100000:1.0:1789760716.175486:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508909312 00000100:00100000:1.0:1789760716.175489:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508909312 00000100:00100000:1.0:1789760716.175526:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 870 00000100:00100000:1.0:1789760716.175533:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508909312:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760716.175561:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508909312:12345-0@lo:400:kworker.0 Request processed in 30us (389us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760716.175568:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 870 00000100:00100000:0.0:1789760716.176052:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760716.176104:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c10e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508909440:0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176128:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760716.176146:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c13100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508909952:0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176167:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760716.176183:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df02c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508910464:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760716.176203:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760716.176235:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c11c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508909312:0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176263:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508909824 00000100:00100000:0.0:1789760716.176267:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508909824 00000100:00100000:0.0:1789760716.176314:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508910336 00000100:00100000:0.0:1789760716.176318:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508910336 00000100:00100000:0.0:1789760716.176338:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508909952 00000100:00100000:0.0:1789760716.176341:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508909952 00000100:00100000:0.0:1789760716.176356:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 306 00000100:00100000:0.0:1789760716.176364:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508909824:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176403:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508909824:12345-0@lo:400:kworker.0 Request processed in 39us (1183us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760716.176410:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 306 00000100:00100000:0.0:1789760716.176422:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 307 00000100:00100000:0.0:1789760716.176430:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508910336:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176455:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508910336:12345-0@lo:400:kworker.0 Request processed in 27us (1199us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760716.176461:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 307 00000100:00100000:0.0:1789760716.176471:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 308 00000100:00100000:0.0:1789760716.176478:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508909952:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176503:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508909952:12345-0@lo:400:kworker.0 Request processed in 26us (335us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760716.176521:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 308 00000100:00100000:0.0:1789760716.176542:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508909440 00000100:00100000:0.0:1789760716.176546:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508909440 00000100:00100000:0.0:1789760716.176566:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 420 00000100:00100000:0.0:1789760716.176573:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508909440:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176612:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508909440:12345-0@lo:400:kworker.0 Request processed in 39us (482us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760716.176619:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 420 00000100:00100000:0.0:1789760716.176657:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c10e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508909440:0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176667:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c13100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508909952:0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176675:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c13480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508909824:0@lo:400:kworker.0 00000100:00100000:0.0:1789760716.176684:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df82d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508910336:0@lo:400:kworker.0 00000100:00100000:3.0:1789760716.178589:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508910464 00000100:00100000:3.0:1789760716.178602:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508910464 00000100:00100000:3.0:1789760716.178644:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 361 00000100:00100000:3.0:1789760716.178652:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508910464:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:3.0:1789760716.178675:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:3.0:1789760716.178743:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508910464:12345-0@lo:13:osp-pre-1-1.0 Request processed in 91us (2538us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760716.178751:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 361 00000100:00100000:3.0:1789760716.178798:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df02c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508910464:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760716.179300:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508908928 00000100:00100000:2.0:1789760716.179306:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508908928 00000100:00100000:2.0:1789760716.179348:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 468 00000100:00100000:2.0:1789760716.179355:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4d2910850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508908928:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760716.179396:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4d2910850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508908928:12345-0@lo:400:kworker.0 Request processed in 42us (3345us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760716.179404:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 468 00000100:00080000:2.0:1789760716.180737:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:2.0:1789760716.180747:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:2.0:1789760716.180750:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760716.180753:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:2.0:1789760716.180794:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c12300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508909184:0@lo:400:kworker.0 00000100:00100000:2.0:1789760716.180809:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c10a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508909696:0@lo:400:kworker.0 00000100:00100000:2.0:1789760716.180821:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c13b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508910208:0@lo:400:kworker.0 00000100:00100000:2.0:1789760716.180832:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c12680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508909056:0@lo:400:kworker.0 00000100:00100000:2.0:1789760716.180842:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c12a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508909568:0@lo:400:kworker.0 00000100:00100000:2.0:1789760716.180852:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c11180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508910080:0@lo:400:kworker.0 00000100:00100000:3.0:1789760716.181792:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c11500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508908928:0@lo:400:kworker.0 00002000:00010000:2.0:1789760716.240894:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19353600 -> 19357696 00002000:00010000:2.0:1789760716.240902:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760715 -> 1789760716 00002000:00010000:2.0:1789760716.240905:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760715 -> 1789760716 00010000:00010000:2.0:1789760716.240918:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19353600->19357695] (req 19353600->19357695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.240931:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19353600->19357695] (req 19353600->19357695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.240946:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19353600->19357695] (req 19353600->19357695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.240954:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760716.240959:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19353600->19357695] (req 19353600->19357695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.240968:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19353600->19357695] (req 19353600->19357695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.240987:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19353600->19357695] (req 19353600->19357695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.240995:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207af0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19353600->19357695] (req 19353600->19357695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760716.241110:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484166400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 71828us (72143us total) trans 4294972043 rc 0/0 00000100:00100000:2.0:1789760716.241121:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4754 00000100:00100000:2.0:1789760716.241131:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705419990562 time 705500197634 state 0x2 ref 0 00000001:00080000:1.0:1789760716.241214:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972043 is committed 00000100:00100000:1.0:1789760716.247774:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760716.248788:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484166528 00000100:00100000:0.0:1789760716.248799:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484166528 00000100:00100000:0.0:1789760716.248868:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705419990562 time 705500197634 state 0x0 ref 1 00000100:00100000:0.0:1789760716.248885:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705500458050 time 705500197634 state 0x1 ref 1 00000100:00100000:0.0:1789760716.248919:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705500458050 time 705579850518 state 0x0 ref 1 00000100:00100000:0.0:1789760716.248933:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484166528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760716.248983:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19357696, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:0.0:1789760716.248992:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.249008:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:0.0:1789760716.249013:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760716.249019:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19357696->19361791] (req 19357696->19361791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.249034:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19357696->19361791] (req 19357696->19361791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760716.369129:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19357696 -> 19361792 00010000:00010000:0.0:1789760716.369149:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19357696->19361791] (req 19357696->19361791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.369168:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19357696->19361791] (req 19357696->19361791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.369178:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19357696->19361791] (req 19357696->19361791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.369185:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760716.369189:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19357696->19361791] (req 19357696->19361791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.369198:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19357696->19361791] (req 19357696->19361791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.369219:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19357696->19361791] (req 19357696->19361791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.369227:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207af7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19357696->19361791] (req 19357696->19361791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760716.369325:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484166528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 120389us (121556us total) trans 4294972044 rc 0/0 00000100:00100000:0.0:1789760716.369336:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4755 00000100:00100000:0.0:1789760716.369346:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705500458050 time 705579850518 state 0x2 ref 0 00000001:00080000:0.0:1789760716.369393:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972044 is committed 00000100:00100000:0.0:1789760716.375780:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760716.375849:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484166656 00000100:00100000:3.0:1789760716.375858:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484166656 00000100:00100000:3.0:1789760716.375910:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705500458050 time 705579850518 state 0x0 ref 1 00000100:00100000:3.0:1789760716.375925:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705580110934 time 705579850518 state 0x1 ref 1 00000100:00100000:3.0:1789760716.375955:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705580110934 time 705706885929 state 0x0 ref 1 00000100:00100000:3.0:1789760716.375966:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7070850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484166656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760716.376009:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19361792, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:3.0:1789760716.376017:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.376031:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:3.0:1789760716.376036:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760716.376042:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19361792->19365887] (req 19361792->19365887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.376056:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19361792->19365887] (req 19361792->19365887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760716.423417:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972045 is committed 00002000:00010000:3.0:1789760716.423491:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19361792 -> 19365888 00010000:00010000:3.0:1789760716.423513:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19361792->19365887] (req 19361792->19365887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.423525:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19361792->19365887] (req 19361792->19365887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.423531:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19361792->19365887] (req 19361792->19365887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.423536:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760716.423539:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19361792->19365887] (req 19361792->19365887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.423545:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19361792->19365887] (req 19361792->19365887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.423559:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19361792->19365887] (req 19361792->19365887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.423564:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207afe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19361792->19365887] (req 19361792->19365887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760716.423724:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7070850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484166656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47756us (47950us total) trans 4294972045 rc 0/0 00000100:00100000:3.0:1789760716.423733:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4756 00000100:00100000:3.0:1789760716.423741:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705580110934 time 705706885929 state 0x2 ref 0 00000100:00100000:0.0:1789760716.426374:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760716.426454:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484166784 00000100:00100000:3.0:1789760716.426463:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484166784 00000100:00100000:3.0:1789760716.426517:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705580110934 time 705706885929 state 0x0 ref 1 00000100:00100000:3.0:1789760716.426532:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705707146345 time 705706885929 state 0x1 ref 1 00000100:00100000:3.0:1789760716.426560:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705707146345 time 705757491006 state 0x0 ref 1 00000100:00100000:3.0:1789760716.426579:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484166784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760716.426635:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19365888, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:3.0:1789760716.426644:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.426658:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:3.0:1789760716.426662:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760716.426668:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19365888->19369983] (req 19365888->19369983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.426681:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19365888->19369983] (req 19365888->19369983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760716.455087:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972046 is committed 00002000:00010000:3.0:1789760716.455174:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19365888 -> 19369984 00010000:00010000:3.0:1789760716.455188:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19365888->19369983] (req 19365888->19369983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.455200:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19365888->19369983] (req 19365888->19369983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.455209:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19365888->19369983] (req 19365888->19369983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.455216:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760716.455221:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19365888->19369983] (req 19365888->19369983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.455231:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19365888->19369983] (req 19365888->19369983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.455248:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19365888->19369983] (req 19365888->19369983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.455255:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b05 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19365888->19369983] (req 19365888->19369983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760716.455342:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484166784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28761us (28973us total) trans 4294972046 rc 0/0 00000100:00100000:3.0:1789760716.455355:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4757 00000100:00100000:3.0:1789760716.455364:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705707146345 time 705757491006 state 0x2 ref 0 00000100:00100000:1.0:1789760716.458871:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760716.458926:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484166912 00000100:00100000:2.0:1789760716.458936:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484166912 00000100:00100000:2.0:1789760716.458991:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705707146345 time 705757491006 state 0x0 ref 1 00000100:00100000:2.0:1789760716.459111:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705757751422 time 705757491006 state 0x1 ref 1 00000100:00100000:2.0:1789760716.459143:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705757751422 time 705790074105 state 0x0 ref 1 00000100:00100000:0.0:1789760716.459205:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484166912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760716.459254:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19369984, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:0.0:1789760716.459264:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.459279:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:0.0:1789760716.459293:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760716.459300:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19369984->19374079] (req 19369984->19374079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.459313:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19369984->19374079] (req 19369984->19374079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760716.525671:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972047 is committed 00002000:00010000:0.0:1789760716.525784:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19369984 -> 19374080 00010000:00010000:0.0:1789760716.525798:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19369984->19374079] (req 19369984->19374079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.525810:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19369984->19374079] (req 19369984->19374079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.525819:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19369984->19374079] (req 19369984->19374079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.525826:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760716.525831:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19369984->19374079] (req 19369984->19374079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.525841:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19369984->19374079] (req 19369984->19374079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.525860:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19369984->19374079] (req 19369984->19374079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.525869:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207b0c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19369984->19374079] (req 19369984->19374079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760716.525965:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484166912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66806us (67099us total) trans 4294972047 rc 0/0 00000100:00100000:0.0:1789760716.525980:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4758 00000100:00100000:0.0:1789760716.525990:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705757751422 time 705790074105 state 0x2 ref 0 00000100:00100000:0.0:1789760716.534299:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760716.534456:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484167040 00000100:00100000:2.0:1789760716.534470:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484167040 00000100:00100000:2.0:1789760716.534534:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705757751422 time 705790074105 state 0x0 ref 1 00000100:00100000:2.0:1789760716.534549:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705790334521 time 705790074105 state 0x1 ref 1 00000100:00100000:2.0:1789760716.534581:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705790334521 time 705865511692 state 0x0 ref 1 00000100:00100000:2.0:1789760716.534594:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484167040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760716.534641:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19374080, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:2.0:1789760716.534649:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.534664:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:2.0:1789760716.534668:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760716.534675:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19374080->19378175] (req 19374080->19378175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.534689:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19374080->19378175] (req 19374080->19378175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760716.606132:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972048 is committed 00002000:00010000:2.0:1789760716.606244:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19374080 -> 19378176 00010000:00010000:2.0:1789760716.606261:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19374080->19378175] (req 19374080->19378175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.606275:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19374080->19378175] (req 19374080->19378175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.606297:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19374080->19378175] (req 19374080->19378175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.606305:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760716.606310:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19374080->19378175] (req 19374080->19378175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.606320:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19374080->19378175] (req 19374080->19378175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.606341:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19374080->19378175] (req 19374080->19378175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.606350:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b13 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19374080->19378175] (req 19374080->19378175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760716.606482:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484167040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 71885us (72198us total) trans 4294972048 rc 0/0 00000100:00100000:2.0:1789760716.606493:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4759 00000100:00100000:2.0:1789760716.606515:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705790334521 time 705865511692 state 0x2 ref 0 00000100:00100000:2.0:1789760716.610568:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760716.610873:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484167168 00000100:00100000:3.0:1789760716.610883:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484167168 00000100:00100000:3.0:1789760716.610936:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705790334521 time 705865511692 state 0x0 ref 1 00000100:00100000:3.0:1789760716.610951:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705865772108 time 705865511692 state 0x1 ref 1 00000100:00100000:3.0:1789760716.610982:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705865772108 time 705941913566 state 0x0 ref 1 00000100:00100000:3.0:1789760716.610995:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484167168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760716.611043:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19378176, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:3.0:1789760716.611051:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.611107:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=705s v=(0 0) 00010000:00010000:3.0:1789760716.611112:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760716.611118:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19378176->19382271] (req 19378176->19382271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.611133:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19378176->19382271] (req 19378176->19382271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760716.654609:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972049 is committed 00002000:00010000:0.0:1789760716.656876:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19378176 -> 19382272 00010000:00010000:0.0:1789760716.656897:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19378176->19382271] (req 19378176->19382271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.656911:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19378176->19382271] (req 19378176->19382271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.656920:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19378176->19382271] (req 19378176->19382271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.656933:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760716.656938:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19378176->19382271] (req 19378176->19382271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.656947:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19378176->19382271] (req 19378176->19382271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.656967:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19378176->19382271] (req 19378176->19382271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.656974:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207b1a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19378176->19382271] (req 19378176->19382271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760716.657111:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484167168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46114us (46548us total) trans 4294972049 rc 0/0 00000100:00100000:0.0:1789760716.657121:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4760 00000100:00100000:0.0:1789760716.657131:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705865772108 time 705941913566 state 0x2 ref 0 00000100:00100000:0.0:1789760716.673458:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760716.673620:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484167296 00000100:00100000:3.0:1789760716.673630:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484167296 00000100:00100000:3.0:1789760716.673770:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705865772108 time 705941913566 state 0x0 ref 1 00000100:00100000:3.0:1789760716.673851:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705942173982 time 705941913566 state 0x1 ref 1 00000100:00100000:3.0:1789760716.673884:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705942173982 time 706004815184 state 0x0 ref 1 00000100:00100000:3.0:1789760716.673898:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484167296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760716.673949:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19382272, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:3.0:1789760716.673958:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.673974:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:3.0:1789760716.673979:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760716.673985:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19382272->19386367] (req 19382272->19386367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.674000:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19382272->19386367] (req 19382272->19386367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.685466:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760716.712796:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972050 is committed 00002000:00010000:2.0:1789760716.712877:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19382272 -> 19386368 00010000:00010000:2.0:1789760716.712893:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19382272->19386367] (req 19382272->19386367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.712905:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19382272->19386367] (req 19382272->19386367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.712912:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19382272->19386367] (req 19382272->19386367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.712919:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760716.712924:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19382272->19386367] (req 19382272->19386367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.712933:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19382272->19386367] (req 19382272->19386367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.712953:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19382272->19386367] (req 19382272->19386367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.712960:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207b21 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19382272->19386367] (req 19382272->19386367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760716.713104:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484167296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39204us (39649us total) trans 4294972050 rc 0/0 00000100:00100000:2.0:1789760716.713114:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4761 00000100:00100000:2.0:1789760716.713124:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705942173982 time 706004815184 state 0x2 ref 0 00000100:00100000:1.0:1789760716.716023:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760716.716158:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484167424 00000100:00100000:0.0:1789760716.716168:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484167424 00000100:00100000:0.0:1789760716.716224:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 705942173982 time 706004815184 state 0x0 ref 1 00000100:00100000:0.0:1789760716.716238:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706005075600 time 706004815184 state 0x1 ref 1 00000100:00100000:0.0:1789760716.716270:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706005075600 time 706047201492 state 0x0 ref 1 00000100:00100000:0.0:1789760716.716282:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484167424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760716.716326:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19386368, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:0.0:1789760716.716335:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.716350:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:0.0:1789760716.716354:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760716.716361:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19386368->19390463] (req 19386368->19390463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.716375:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19386368->19390463] (req 19386368->19390463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760716.767187:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972051 is committed 00002000:00010000:0.0:1789760716.767426:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19386368 -> 19390464 00010000:00010000:0.0:1789760716.767448:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19386368->19390463] (req 19386368->19390463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.767462:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19386368->19390463] (req 19386368->19390463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.767473:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19386368->19390463] (req 19386368->19390463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.767487:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760716.767493:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19386368->19390463] (req 19386368->19390463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.767515:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19386368->19390463] (req 19386368->19390463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.767541:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19386368->19390463] (req 19386368->19390463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.767549:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b28 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19386368->19390463] (req 19386368->19390463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760716.767683:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484167424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51396us (51665us total) trans 4294972051 rc 0/0 00000100:00100000:0.0:1789760716.767696:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4762 00000100:00100000:0.0:1789760716.767708:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706005075600 time 706047201492 state 0x2 ref 0 00000100:00100000:0.0:1789760716.774774:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760716.774973:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484167552 00000100:00100000:2.0:1789760716.774988:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484167552 00000100:00100000:2.0:1789760716.775106:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706005075600 time 706047201492 state 0x0 ref 1 00000100:00100000:2.0:1789760716.775125:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706047461908 time 706047201492 state 0x1 ref 1 00000100:00100000:2.0:1789760716.775163:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706047461908 time 706106093507 state 0x0 ref 1 00000100:00100000:2.0:1789760716.775176:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484167552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760716.775240:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19390464, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:2.0:1789760716.775251:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.775271:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:2.0:1789760716.775276:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760716.775283:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19390464->19394559] (req 19390464->19394559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.775299:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19390464->19394559] (req 19390464->19394559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760716.843193:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972052 is committed 00002000:00010000:0.0:1789760716.843279:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19390464 -> 19394560 00010000:00010000:0.0:1789760716.843309:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19390464->19394559] (req 19390464->19394559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.843323:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19390464->19394559] (req 19390464->19394559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.843334:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19390464->19394559] (req 19390464->19394559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.843343:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760716.843349:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19390464->19394559] (req 19390464->19394559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.843359:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19390464->19394559] (req 19390464->19394559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.843385:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19390464->19394559] (req 19390464->19394559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760716.843393:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207b2f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19390464->19394559] (req 19390464->19394559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760716.843528:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484167552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68347us (68764us total) trans 4294972052 rc 0/0 00000100:00100000:0.0:1789760716.843540:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4763 00000100:00100000:0.0:1789760716.843553:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706047461908 time 706106093507 state 0x2 ref 0 00000100:00100000:1.0:1789760716.856640:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760716.856906:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484167680 00000100:00100000:3.0:1789760716.856920:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484167680 00000100:00100000:3.0:1789760716.856995:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706047461908 time 706106093507 state 0x0 ref 1 00000100:00100000:3.0:1789760716.857014:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706106353923 time 706106093507 state 0x1 ref 1 00000100:00100000:3.0:1789760716.857046:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706106353923 time 706187976021 state 0x0 ref 1 00000100:00100000:3.0:1789760716.857060:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484167680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760716.857164:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19394560, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:3.0:1789760716.857175:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.857195:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:3.0:1789760716.857200:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760716.857208:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19394560->19398655] (req 19394560->19398655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760716.857225:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19394560->19398655] (req 19394560->19398655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760716.937514:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19394560 -> 19398656 00010000:00010000:2.0:1789760716.937541:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19394560->19398655] (req 19394560->19398655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.937560:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19394560->19398655] (req 19394560->19398655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.937571:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19394560->19398655] (req 19394560->19398655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.937579:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760716.937584:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19394560->19398655] (req 19394560->19398655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.937711:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19394560->19398655] (req 19394560->19398655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.937738:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19394560->19398655] (req 19394560->19398655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760716.937747:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207b36 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19394560->19398655] (req 19394560->19398655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760716.937962:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484167680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 80890us (81323us total) trans 4294972053 rc 0/0 00000100:00100000:2.0:1789760716.937976:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4764 00000100:00100000:2.0:1789760716.937989:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706106353923 time 706187976021 state 0x2 ref 0 00000100:00100000:1.0:1789760716.945662:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760716.945717:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484167808 00000100:00100000:1.0:1789760716.945730:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484167808 00000100:00100000:1.0:1789760716.945800:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706106353923 time 706187976021 state 0x0 ref 1 00000100:00100000:1.0:1789760716.945888:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706188236437 time 706187976021 state 0x1 ref 1 00000100:00100000:1.0:1789760716.945917:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706188236437 time 706276848480 state 0x0 ref 1 00000100:00100000:1.0:1789760716.945931:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484167808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760716.945994:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19398656, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:1.0:1789760716.946005:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760716.946026:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:1.0:1789760716.946031:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760716.946038:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19398656->19402751] (req 19398656->19402751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760716.946054:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19398656->19402751] (req 19398656->19402751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760716.951029:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972053 is committed 00000001:00080000:0.0:1789760717.014435:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972054 is committed 00002000:00010000:1.0:1789760717.014720:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19398656 -> 19402752 00010000:00010000:1.0:1789760717.014740:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19398656->19402751] (req 19398656->19402751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.014755:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19398656->19402751] (req 19398656->19402751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.014764:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19398656->19402751] (req 19398656->19402751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.014772:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760717.014778:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19398656->19402751] (req 19398656->19402751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.014788:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19398656->19402751] (req 19398656->19402751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.014813:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19398656->19402751] (req 19398656->19402751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.014822:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b3d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19398656->19402751] (req 19398656->19402751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.014945:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484167808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69009us (69291us total) trans 4294972054 rc 0/0 00000100:00100000:1.0:1789760717.014962:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4765 00000100:00100000:1.0:1789760717.014975:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706188236437 time 706276848480 state 0x2 ref 0 00000100:00100000:0.0:1789760717.021612:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.021713:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484167936 00000100:00100000:1.0:1789760717.021726:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484167936 00000100:00100000:1.0:1789760717.021799:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706188236437 time 706276848480 state 0x0 ref 1 00000100:00100000:1.0:1789760717.021818:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706277108896 time 706276848480 state 0x1 ref 1 00000100:00100000:1.0:1789760717.021856:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706277108896 time 706352786619 state 0x0 ref 1 00000100:00100000:1.0:1789760717.021870:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484167936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760717.021935:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19402752, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:1.0:1789760717.021946:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.021977:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:1.0:1789760717.021982:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760717.021990:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19402752->19406847] (req 19402752->19406847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.022006:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19402752->19406847] (req 19402752->19406847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.030048:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.030201:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760717.033660:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484168192 00000100:00100000:2.0:1789760717.033673:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484168192 00000100:00100000:3.0:1789760717.033873:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484168448 00000100:00100000:3.0:1789760717.033878:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484168448 00000100:00100000:1.0:1789760717.033946:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.033975:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.034008:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760717.034127:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484168320 00000100:00100000:3.0:1789760717.034131:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484168320 00000100:00100000:1.0:1789760717.034145:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484168576 00000100:00100000:1.0:1789760717.034152:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484168576 00000100:00100000:3.0:1789760717.034159:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 871 00000100:00100000:3.0:1789760717.034171:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484168448:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760717.034187:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 872 00000100:00100000:1.0:1789760717.034196:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484168320:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760717.034239:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484168448:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 71us (4039us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760717.034246:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 871 00000100:00100000:3.0:1789760717.034275:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 873 00000100:00100000:0.0:1789760717.034278:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484168064 00000100:00100000:3.0:1789760717.034284:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82d880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484168576:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760717.034285:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484168064 00000100:00100000:0.0:1789760717.034328:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 469 00000100:00100000:0.0:1789760717.034337:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b4050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484168064:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760717.034447:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484168320:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 252us (470us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760717.034455:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 872 00000100:00100000:3.0:1789760717.034524:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82d880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484168576:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 241us (514us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760717.034532:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 873 00000100:00080000:3.0:1789760717.034553:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:3.0:1789760717.034558:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:0.0:1789760717.035981:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b4050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484168064:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 1640us (2033us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760717.035988:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 469 00000100:00100000:3.0:1789760717.036435:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 874 00000100:00100000:3.0:1789760717.036445:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82d180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484168192:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760717.036495:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82d180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484168192:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 53us (6457us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760717.036516:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 874 00000100:00080000:3.0:1789760717.036533:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000001:00080000:3.0:1789760717.060873:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972055 is committed 00002000:00010000:1.0:1789760717.061115:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19402752 -> 19406848 00010000:00010000:1.0:1789760717.061131:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19402752->19406847] (req 19402752->19406847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.061204:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19402752->19406847] (req 19402752->19406847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.061215:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19402752->19406847] (req 19402752->19406847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.061222:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760717.061227:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19402752->19406847] (req 19402752->19406847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.061236:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19402752->19406847] (req 19402752->19406847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.061262:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19402752->19406847] (req 19402752->19406847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.061270:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b44 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19402752->19406847] (req 19402752->19406847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.061359:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484167936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39486us (39767us total) trans 4294972055 rc 0/0 00000100:00100000:1.0:1789760717.061368:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4766 00000100:00100000:1.0:1789760717.061378:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706277108896 time 706352786619 state 0x2 ref 0 00000100:00100000:0.0:1789760717.064723:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.064993:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484168704 00000100:00100000:1.0:1789760717.065003:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484168704 00000100:00100000:1.0:1789760717.066661:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706277108896 time 706352786619 state 0x0 ref 1 00000100:00100000:1.0:1789760717.066677:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706353047035 time 706352786619 state 0x1 ref 1 00000100:00100000:1.0:1789760717.066713:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706353047035 time 706397643358 state 0x0 ref 1 00000100:00100000:1.0:1789760717.066727:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484168704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760717.066773:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19406848, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:1.0:1789760717.066782:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.066798:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:1.0:1789760717.066803:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760717.066809:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19406848->19410943] (req 19406848->19410943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.066824:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19406848->19410943] (req 19406848->19410943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760717.110967:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972056 is committed 00002000:00010000:1.0:1789760717.111190:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19406848 -> 19410944 00010000:00010000:1.0:1789760717.111205:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19406848->19410943] (req 19406848->19410943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.111218:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19406848->19410943] (req 19406848->19410943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.111227:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19406848->19410943] (req 19406848->19410943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.111235:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760717.111240:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19406848->19410943] (req 19406848->19410943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.111251:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19406848->19410943] (req 19406848->19410943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.111272:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19406848->19410943] (req 19406848->19410943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.111298:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207b4b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19406848->19410943] (req 19406848->19410943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.111396:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484168704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44668us (46677us total) trans 4294972056 rc 0/0 00000100:00100000:1.0:1789760717.111406:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4767 00000100:00100000:1.0:1789760717.111417:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706353047035 time 706397643358 state 0x2 ref 0 00000100:00100000:3.0:1789760717.114226:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760717.116943:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484168832 00000100:00100000:0.0:1789760717.116960:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484168832 00000100:00100000:1.0:1789760717.117364:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706353047035 time 706397643358 state 0x0 ref 1 00000100:00100000:1.0:1789760717.117382:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706397903774 time 706397643358 state 0x1 ref 1 00000100:00100000:1.0:1789760717.117454:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706397903774 time 706448384162 state 0x0 ref 1 00000100:00100000:1.0:1789760717.117469:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484168832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760717.117543:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19410944, mtime: 0x6aad94cc, atime: 0x0, ctime: 0x6aad94cc, blocks: 0x9807 00010000:00010000:1.0:1789760717.117553:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.117569:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:1.0:1789760717.117574:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760717.117580:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19410944->19415039] (req 19410944->19415039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.117593:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19410944->19415039] (req 19410944->19415039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760717.147728:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972057 is committed 00002000:00010000:0.0:1789760717.147804:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19410944 -> 19415040 00002000:00010000:0.0:1789760717.147808:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760716 -> 1789760717 00002000:00010000:0.0:1789760717.147811:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760716 -> 1789760717 00010000:00010000:0.0:1789760717.147824:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19410944->19415039] (req 19410944->19415039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.147837:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19410944->19415039] (req 19410944->19415039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.147846:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19410944->19415039] (req 19410944->19415039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.147853:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760717.147858:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19410944->19415039] (req 19410944->19415039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.147873:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19410944->19415039] (req 19410944->19415039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.147892:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19410944->19415039] (req 19410944->19415039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.147900:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396207b52 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19410944->19415039] (req 19410944->19415039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760717.147998:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484168832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30528us (33777us total) trans 4294972057 rc 0/0 00000100:00100000:0.0:1789760717.148008:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4768 00000100:00100000:0.0:1789760717.148017:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706397903774 time 706448384162 state 0x2 ref 0 00000100:00100000:2.0:1789760717.151744:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.151853:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484168960 00000100:00100000:1.0:1789760717.151869:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484168960 00000100:00100000:1.0:1789760717.151931:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706397903774 time 706448384162 state 0x0 ref 1 00000100:00100000:1.0:1789760717.151946:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706448644578 time 706448384162 state 0x1 ref 1 00000100:00100000:1.0:1789760717.151985:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706448644578 time 706482910369 state 0x0 ref 1 00000100:00100000:1.0:1789760717.151999:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484168960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760717.152049:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19415040, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:1.0:1789760717.152059:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.152112:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:1.0:1789760717.152116:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760717.152122:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19415040->19419135] (req 19415040->19419135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.152137:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19415040->19419135] (req 19415040->19419135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760717.220938:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972058 is committed 00002000:00010000:0.0:1789760717.221194:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19415040 -> 19419136 00010000:00010000:0.0:1789760717.221214:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19415040->19419135] (req 19415040->19419135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.221230:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19415040->19419135] (req 19415040->19419135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.221240:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19415040->19419135] (req 19415040->19419135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.221247:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760717.221253:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19415040->19419135] (req 19415040->19419135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.221262:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19415040->19419135] (req 19415040->19419135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.221296:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19415040->19419135] (req 19415040->19419135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.221304:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207b59 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19415040->19419135] (req 19415040->19419135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760717.221429:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484168960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69424us (69691us total) trans 4294972058 rc 0/0 00000100:00100000:0.0:1789760717.221441:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4769 00000100:00100000:0.0:1789760717.221452:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706448644578 time 706482910369 state 0x2 ref 0 00000100:00100000:1.0:1789760717.224750:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760717.224894:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484169088 00000100:00100000:0.0:1789760717.224907:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484169088 00000100:00100000:0.0:1789760717.224987:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706448644578 time 706482910369 state 0x0 ref 1 00000100:00100000:0.0:1789760717.225007:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706483170785 time 706482910369 state 0x1 ref 1 00000100:00100000:0.0:1789760717.225049:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706483170785 time 706555980300 state 0x0 ref 1 00000100:00100000:0.0:1789760717.225100:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484169088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760717.225171:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19419136, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:0.0:1789760717.225184:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.225204:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:0.0:1789760717.225210:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760717.225218:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19419136->19423231] (req 19419136->19423231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.225234:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19419136->19423231] (req 19419136->19423231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760717.260420:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972059 is committed 00002000:00010000:2.0:1789760717.260907:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19419136 -> 19423232 00010000:00010000:2.0:1789760717.260926:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19419136->19423231] (req 19419136->19423231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.260941:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19419136->19423231] (req 19419136->19423231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.260951:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19419136->19423231] (req 19419136->19423231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.260959:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760717.260965:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19419136->19423231] (req 19419136->19423231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.260975:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19419136->19423231] (req 19419136->19423231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.261000:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19419136->19423231] (req 19419136->19423231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.261011:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207b60 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19419136->19423231] (req 19419136->19423231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760717.261177:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484169088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36109us (36435us total) trans 4294972059 rc 0/0 00000100:00100000:2.0:1789760717.261188:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4770 00000100:00100000:2.0:1789760717.261201:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706483170785 time 706555980300 state 0x2 ref 0 00000100:00100000:3.0:1789760717.264516:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760717.264765:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484169216 00000100:00100000:2.0:1789760717.264780:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484169216 00000100:00100000:2.0:1789760717.264849:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706483170785 time 706555980300 state 0x0 ref 1 00000100:00100000:2.0:1789760717.264866:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706556240716 time 706555980300 state 0x1 ref 1 00000100:00100000:2.0:1789760717.264897:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706556240716 time 706595828236 state 0x0 ref 1 00000100:00100000:2.0:1789760717.264910:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484169216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760717.264970:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19423232, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:2.0:1789760717.264980:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.264999:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:2.0:1789760717.265004:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760717.265011:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19423232->19427327] (req 19423232->19427327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.265027:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19423232->19427327] (req 19423232->19427327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760717.298252:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972060 is committed 00002000:00010000:0.0:1789760717.298404:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19423232 -> 19427328 00010000:00010000:0.0:1789760717.298426:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19423232->19427327] (req 19423232->19427327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.298442:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19423232->19427327] (req 19423232->19427327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.298454:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19423232->19427327] (req 19423232->19427327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.298803:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760717.298809:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19423232->19427327] (req 19423232->19427327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.298823:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19423232->19427327] (req 19423232->19427327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.298856:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19423232->19427327] (req 19423232->19427327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.298865:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207b67 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19423232->19427327] (req 19423232->19427327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760717.298992:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484169216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34077us (34493us total) trans 4294972060 rc 0/0 00000100:00100000:0.0:1789760717.299005:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4771 00000100:00100000:0.0:1789760717.299017:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706556240716 time 706595828236 state 0x2 ref 0 00000100:00100000:1.0:1789760717.302043:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760717.302451:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484169344 00000100:00100000:0.0:1789760717.302536:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484169344 00000100:00100000:0.0:1789760717.302743:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706556240716 time 706595828236 state 0x0 ref 1 00000100:00100000:0.0:1789760717.302763:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706596088652 time 706595828236 state 0x1 ref 1 00000100:00100000:0.0:1789760717.302838:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706596088652 time 706633767692 state 0x0 ref 1 00000100:00100000:0.0:1789760717.302855:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484169344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760717.302922:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19427328, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:0.0:1789760717.302933:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.302952:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:0.0:1789760717.302958:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760717.302965:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19427328->19431423] (req 19427328->19431423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.302982:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19427328->19431423] (req 19427328->19431423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760717.346054:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972061 is committed 00002000:00010000:1.0:1789760717.346147:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19427328 -> 19431424 00010000:00010000:1.0:1789760717.346261:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19427328->19431423] (req 19427328->19431423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.346276:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19427328->19431423] (req 19427328->19431423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.346369:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19427328->19431423] (req 19427328->19431423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.346378:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760717.346384:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19427328->19431423] (req 19427328->19431423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.346395:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19427328->19431423] (req 19427328->19431423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.346421:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19427328->19431423] (req 19427328->19431423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.346429:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207b6e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19427328->19431423] (req 19427328->19431423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.346805:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484169344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43944us (44766us total) trans 4294972061 rc 0/0 00000100:00100000:1.0:1789760717.346819:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4772 00000100:00100000:1.0:1789760717.346831:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706596088652 time 706633767692 state 0x2 ref 0 00000100:00100000:1.0:1789760717.353534:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760717.353749:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484169472 00000100:00100000:0.0:1789760717.353764:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484169472 00000100:00100000:0.0:1789760717.353852:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706596088652 time 706633767692 state 0x0 ref 1 00000100:00100000:0.0:1789760717.353871:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706634028108 time 706633767692 state 0x1 ref 1 00000100:00100000:0.0:1789760717.353913:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706634028108 time 706684843356 state 0x0 ref 1 00000100:00100000:0.0:1789760717.353928:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484169472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760717.353999:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19431424, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:0.0:1789760717.354010:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.354030:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:0.0:1789760717.354035:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760717.354042:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19431424->19435519] (req 19431424->19435519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.354059:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19431424->19435519] (req 19431424->19435519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760717.391893:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19431424 -> 19435520 00010000:00010000:0.0:1789760717.391922:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19431424->19435519] (req 19431424->19435519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.391938:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19431424->19435519] (req 19431424->19435519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.391948:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19431424->19435519] (req 19431424->19435519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.391957:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760717.391962:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19431424->19435519] (req 19431424->19435519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.391973:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19431424->19435519] (req 19431424->19435519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.391998:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19431424->19435519] (req 19431424->19435519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.392007:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b75 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19431424->19435519] (req 19431424->19435519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760717.392173:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484169472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38242us (38651us total) trans 4294972062 rc 0/0 00000100:00100000:0.0:1789760717.392189:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4773 00000100:00100000:0.0:1789760717.392203:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706634028108 time 706684843356 state 0x2 ref 0 00000001:00080000:3.0:1789760717.392456:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972062 is committed 00000100:00100000:3.0:1789760717.395012:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760717.395154:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484169600 00000100:00100000:2.0:1789760717.395168:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484169600 00000100:00100000:2.0:1789760717.395241:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706634028108 time 706684843356 state 0x0 ref 1 00000100:00100000:2.0:1789760717.395259:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706685103772 time 706684843356 state 0x1 ref 1 00000100:00100000:1.0:1789760717.395356:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706685103772 time 706726284645 state 0x0 ref 1 00000100:00100000:1.0:1789760717.395369:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484169600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760717.395428:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19435520, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:1.0:1789760717.395437:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.395453:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:1.0:1789760717.395457:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760717.395471:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19435520->19439615] (req 19435520->19439615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.395484:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19435520->19439615] (req 19435520->19439615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760717.442253:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19435520 -> 19439616 00010000:00010000:1.0:1789760717.442280:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19435520->19439615] (req 19435520->19439615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.442296:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19435520->19439615] (req 19435520->19439615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.442305:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19435520->19439615] (req 19435520->19439615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.442319:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760717.442325:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19435520->19439615] (req 19435520->19439615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.442335:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19435520->19439615] (req 19435520->19439615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.442362:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19435520->19439615] (req 19435520->19439615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.442372:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b7c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19435520->19439615] (req 19435520->19439615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.442488:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484169600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47114us (47483us total) trans 4294972063 rc 0/0 00000100:00100000:1.0:1789760717.442513:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4774 00000100:00100000:1.0:1789760717.442525:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706685103772 time 706726284645 state 0x2 ref 0 00000001:00080000:3.0:1789760717.442650:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972063 is committed 00000100:00100000:1.0:1789760717.448132:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760717.448385:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484169728 00000100:00100000:2.0:1789760717.448398:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484169728 00000100:00100000:2.0:1789760717.448466:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706685103772 time 706726284645 state 0x0 ref 1 00000100:00100000:2.0:1789760717.448482:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706726545061 time 706726284645 state 0x1 ref 1 00000100:00100000:2.0:1789760717.448520:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706726545061 time 706779451422 state 0x0 ref 1 00000100:00100000:2.0:1789760717.448534:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484169728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760717.448612:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19439616, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:2.0:1789760717.448623:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.448642:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:2.0:1789760717.448646:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760717.448652:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19439616->19443711] (req 19439616->19443711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.448772:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19439616->19443711] (req 19439616->19443711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760717.515098:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972064 is committed 00002000:00010000:1.0:1789760717.515245:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19439616 -> 19443712 00010000:00010000:1.0:1789760717.515266:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19439616->19443711] (req 19439616->19443711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.515292:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19439616->19443711] (req 19439616->19443711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.515308:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19439616->19443711] (req 19439616->19443711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.515316:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760717.515322:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19439616->19443711] (req 19439616->19443711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.515333:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19439616->19443711] (req 19439616->19443711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.515359:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19439616->19443711] (req 19439616->19443711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.515369:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207b83 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19439616->19443711] (req 19439616->19443711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.515637:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484169728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67097us (67512us total) trans 4294972064 rc 0/0 00000100:00100000:1.0:1789760717.515649:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4775 00000100:00100000:1.0:1789760717.515661:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706726545061 time 706779451422 state 0x2 ref 0 00000100:00100000:2.0:1789760717.519043:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760717.519310:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484169856 00000100:00100000:0.0:1789760717.519323:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484169856 00000100:00100000:0.0:1789760717.519405:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706726545061 time 706779451422 state 0x0 ref 1 00000100:00100000:0.0:1789760717.519424:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706779711838 time 706779451422 state 0x1 ref 1 00000100:00100000:0.0:1789760717.519549:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706779711838 time 706850478137 state 0x0 ref 1 00000100:00100000:0.0:1789760717.519566:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484169856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760717.519632:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19443712, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:0.0:1789760717.519643:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.519671:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:0.0:1789760717.519676:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760717.519683:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19443712->19447807] (req 19443712->19447807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.519701:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19443712->19447807] (req 19443712->19447807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760717.597730:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19443712 -> 19447808 00010000:00010000:1.0:1789760717.597760:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19443712->19447807] (req 19443712->19447807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.597783:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19443712->19447807] (req 19443712->19447807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.597793:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19443712->19447807] (req 19443712->19447807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.597802:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760717.597807:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19443712->19447807] (req 19443712->19447807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.597819:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19443712->19447807] (req 19443712->19447807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.597844:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19443712->19447807] (req 19443712->19447807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.597852:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207b8a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19443712->19447807] (req 19443712->19447807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.597972:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484169856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 78404us (78937us total) trans 4294972065 rc 0/0 00000100:00100000:1.0:1789760717.597986:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4776 00000100:00100000:1.0:1789760717.597998:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706779711838 time 706850478137 state 0x2 ref 0 00000001:00080000:3.0:1789760717.599584:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972065 is committed 00000100:00100000:2.0:1789760717.610118:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760717.611443:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484169984 00000100:00100000:0.0:1789760717.611471:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484169984 00000100:00100000:0.0:1789760717.611579:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706779711838 time 706850478137 state 0x0 ref 1 00000100:00100000:0.0:1789760717.611609:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706850738553 time 706850478137 state 0x1 ref 1 00000100:00100000:0.0:1789760717.611647:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706850738553 time 706942577136 state 0x0 ref 1 00000100:00100000:0.0:1789760717.611941:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484169984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760717.612014:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19447808, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:0.0:1789760717.612025:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.612046:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=706s v=(0 0) 00010000:00010000:0.0:1789760717.612051:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760717.612058:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19447808->19451903] (req 19447808->19451903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.612123:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19447808->19451903] (req 19447808->19451903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.708607:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508910592:0@lo:13:osp-pre-0-0.0 00000100:00100000:1.0:1789760717.708727:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760717.708916:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508910592 00000100:00100000:3.0:1789760717.708932:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508910592 00000100:00100000:3.0:1789760717.708986:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 362 00000100:00100000:3.0:1789760717.708996:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508910592:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:3.0:1789760717.709037:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527499264 pending 1703936 free 3838836736 avail 3836739584 00000100:00100000:2.0:1789760717.713365:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760717.713389:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c880ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508910848:0@lo:41:osp-pre-0-1.0 00000100:00100000:2.0:1789760717.713487:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760717.713574:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c880f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508910720:0@lo:13:osp-pre-0-1.0 00000100:00100000:1.0:1789760717.713577:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508910848 00000100:00100000:1.0:1789760717.713594:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508910848 00000100:00100000:2.0:1789760717.713607:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760717.713689:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508910592:12345-0@lo:13:osp-pre-0-0.0 Request processed in 4692us (4960us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760717.713699:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 362 00000100:00100000:3.0:1789760717.713725:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508910720 00000100:00100000:3.0:1789760717.713732:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508910720 00002000:00010000:2.0:1789760717.713818:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19447808 -> 19451904 00010000:00010000:2.0:1789760717.713831:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19447808->19451903] (req 19447808->19451903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.713844:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 421 00010000:00010000:2.0:1789760717.713850:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19447808->19451903] (req 19447808->19451903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.713864:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508910848:12345-0@lo:41:osp-pre-0-1.0 00010000:00010000:2.0:1789760717.713865:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19447808->19451903] (req 19447808->19451903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.713881:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760717.713886:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19447808->19451903] (req 19447808->19451903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.713894:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19447808->19451903] (req 19447808->19451903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.713913:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19447808->19451903] (req 19447808->19451903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.713921:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207b91 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19447808->19451903] (req 19447808->19451903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000004:00000024:1.0:1789760717.713925:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:2.0:1789760717.714016:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484169984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 102354us (103953us total) trans 4294972066 rc 0/0 00000100:00100000:2.0:1789760717.714027:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4777 00000100:00100000:2.0:1789760717.714037:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706850738553 time 706942577136 state 0x2 ref 0 00000100:00100000:1.0:1789760717.714826:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508910848:12345-0@lo:41:osp-pre-0-1.0 Request processed in 966us (1334us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760717.714838:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 421 00000100:00100000:1.0:1789760717.714927:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c880ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508910848:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760717.716948:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 363 00000100:00100000:0.0:1789760717.716977:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508910720:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:0.0:1789760717.717006:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527499264 pending 1703936 free 3838836736 avail 3836739584 00000001:00080000:3.0:1789760717.717015:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972066 is committed 00010000:00010000:1.0:1789760717.717036:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760717.717107:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508910720:12345-0@lo:13:osp-pre-0-1.0 Request processed in 147us (3500us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760717.717110:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508910592:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760717.717115:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 363 00000100:00100000:2.0:1789760717.717578:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c880f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508910720:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760717.725687:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.726798:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484170112 00000100:00100000:1.0:1789760717.726820:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484170112 00000100:00100000:1.0:1789760717.726904:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706850738553 time 706942577136 state 0x0 ref 1 00000100:00100000:1.0:1789760717.726923:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706942837552 time 706942577136 state 0x1 ref 1 00000100:00100000:1.0:1789760717.726962:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706942837552 time 707057892806 state 0x0 ref 1 00000100:00100000:1.0:1789760717.726976:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484170112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760717.727050:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19451904, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:1.0:1789760717.727061:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.727123:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:1.0:1789760717.727129:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760717.727137:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19451904->19455999] (req 19451904->19455999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.727154:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19451904->19455999] (req 19451904->19455999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760717.788805:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972067 is committed 00002000:00010000:1.0:1789760717.788965:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19451904 -> 19456000 00010000:00010000:1.0:1789760717.788981:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19451904->19455999] (req 19451904->19455999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.788994:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19451904->19455999] (req 19451904->19455999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.789003:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19451904->19455999] (req 19451904->19455999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.789011:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760717.789017:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19451904->19455999] (req 19451904->19455999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.789027:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19451904->19455999] (req 19451904->19455999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.789046:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19451904->19455999] (req 19451904->19455999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.789055:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396207b98 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19451904->19455999] (req 19451904->19455999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760717.789164:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484170112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62185us (63481us total) trans 4294972067 rc 0/0 00000100:00100000:1.0:1789760717.789175:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4778 00000100:00100000:1.0:1789760717.789185:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706942837552 time 707057892806 state 0x2 ref 0 00000100:00100000:2.0:1789760717.792452:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.792554:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484170240 00000100:00100000:1.0:1789760717.792564:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484170240 00000100:00100000:1.0:1789760717.792626:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 706942837552 time 707057892806 state 0x0 ref 1 00000100:00100000:1.0:1789760717.792641:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707058153222 time 707057892806 state 0x1 ref 1 00000100:00100000:0.0:1789760717.792744:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707058153222 time 707123673565 state 0x0 ref 1 00000100:00100000:0.0:1789760717.792763:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484170240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760717.792803:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19456000, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:0.0:1789760717.792809:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.792822:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:0.0:1789760717.792825:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760717.792830:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19456000->19460095] (req 19456000->19460095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.792840:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19456000->19460095] (req 19456000->19460095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760717.822771:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972068 is committed 00002000:00010000:0.0:1789760717.823050:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19456000 -> 19460096 00010000:00010000:0.0:1789760717.823694:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19456000->19460095] (req 19456000->19460095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.823709:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19456000->19460095] (req 19456000->19460095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.823719:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19456000->19460095] (req 19456000->19460095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.823726:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760717.823731:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19456000->19460095] (req 19456000->19460095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.823740:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19456000->19460095] (req 19456000->19460095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.823760:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19456000->19460095] (req 19456000->19460095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.823768:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207b9f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19456000->19460095] (req 19456000->19460095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760717.823872:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484170240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31112us (31425us total) trans 4294972068 rc 0/0 00000100:00100000:0.0:1789760717.823882:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4779 00000100:00100000:0.0:1789760717.823891:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707058153222 time 707123673565 state 0x2 ref 0 00000100:00100000:2.0:1789760717.831628:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760717.831704:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484170368 00000100:00100000:0.0:1789760717.831712:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484170368 00000100:00100000:0.0:1789760717.831762:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707058153222 time 707123673565 state 0x0 ref 1 00000100:00100000:0.0:1789760717.831773:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707123933981 time 707123673565 state 0x1 ref 1 00000100:00100000:0.0:1789760717.831798:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707123933981 time 707162730743 state 0x0 ref 1 00000100:00100000:0.0:1789760717.831807:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484170368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760717.831849:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19460096, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:0.0:1789760717.831856:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.831869:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:0.0:1789760717.831873:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760717.831877:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19460096->19464191] (req 19460096->19464191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.831887:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19460096->19464191] (req 19460096->19464191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760717.868140:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972069 is committed 00002000:00010000:0.0:1789760717.868195:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19460096 -> 19464192 00010000:00010000:0.0:1789760717.868209:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19460096->19464191] (req 19460096->19464191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.868223:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19460096->19464191] (req 19460096->19464191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.868232:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19460096->19464191] (req 19460096->19464191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.868239:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760717.868244:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19460096->19464191] (req 19460096->19464191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.868255:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19460096->19464191] (req 19460096->19464191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.868360:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19460096->19464191] (req 19460096->19464191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.868369:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ba6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19460096->19464191] (req 19460096->19464191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760717.868468:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484170368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36655us (36843us total) trans 4294972069 rc 0/0 00000100:00100000:0.0:1789760717.868479:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4780 00000100:00100000:0.0:1789760717.868488:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707123933981 time 707162730743 state 0x2 ref 0 00000100:00100000:1.0:1789760717.870924:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760717.871029:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484170496 00000100:00100000:2.0:1789760717.871038:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484170496 00000100:00100000:2.0:1789760717.871098:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707123933981 time 707162730743 state 0x0 ref 1 00000100:00100000:2.0:1789760717.871110:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707162991159 time 707162730743 state 0x1 ref 1 00000100:00100000:2.0:1789760717.871140:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707162991159 time 707202072714 state 0x0 ref 1 00000100:00100000:2.0:1789760717.871201:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484170496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760717.871238:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19464192, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:2.0:1789760717.871245:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.871256:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:2.0:1789760717.871260:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760717.871264:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19464192->19468287] (req 19464192->19468287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760717.871283:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19464192->19468287] (req 19464192->19468287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760717.909545:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972070 is committed 00002000:00010000:3.0:1789760717.909624:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19464192 -> 19468288 00010000:00010000:3.0:1789760717.909637:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19464192->19468287] (req 19464192->19468287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.909651:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19464192->19468287] (req 19464192->19468287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.909661:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19464192->19468287] (req 19464192->19468287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.909668:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760717.909673:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19464192->19468287] (req 19464192->19468287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.909682:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19464192->19468287] (req 19464192->19468287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.909704:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19464192->19468287] (req 19464192->19468287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.909713:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207bad lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19464192->19468287] (req 19464192->19468287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760717.909956:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484170496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38801us (39035us total) trans 4294972070 rc 0/0 00000100:00100000:3.0:1789760717.909970:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4781 00000100:00100000:3.0:1789760717.909979:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707162991159 time 707202072714 state 0x2 ref 0 00000100:00100000:2.0:1789760717.922244:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.922808:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484170624 00000100:00100000:1.0:1789760717.922819:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484170624 00000100:00100000:1.0:1789760717.922931:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707162991159 time 707202072714 state 0x0 ref 1 00000100:00100000:1.0:1789760717.922948:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707202333130 time 707202072714 state 0x1 ref 1 00000100:00100000:1.0:1789760717.923044:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707202333130 time 707253974486 state 0x0 ref 1 00000100:00100000:1.0:1789760717.923057:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484170624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760717.923144:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19468288, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:1.0:1789760717.923153:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.923169:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:1.0:1789760717.923173:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760717.923178:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19468288->19472383] (req 19468288->19472383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.923190:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19468288->19472383] (req 19468288->19472383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760717.942834:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972071 is committed 00002000:00010000:3.0:1789760717.942913:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19468288 -> 19472384 00010000:00010000:3.0:1789760717.942929:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19468288->19472383] (req 19468288->19472383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.942941:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19468288->19472383] (req 19468288->19472383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.942951:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19468288->19472383] (req 19468288->19472383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.942959:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760717.942963:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19468288->19472383] (req 19468288->19472383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.942972:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19468288->19472383] (req 19468288->19472383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.942991:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19468288->19472383] (req 19468288->19472383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.942999:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bb4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19468288->19472383] (req 19468288->19472383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760717.943239:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484170624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 20180us (20999us total) trans 4294972071 rc 0/0 00000100:00100000:3.0:1789760717.943250:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4782 00000100:00100000:3.0:1789760717.943259:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707202333130 time 707253974486 state 0x2 ref 0 00000100:00100000:3.0:1789760717.946538:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760717.946650:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484170752 00000100:00100000:0.0:1789760717.946660:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484170752 00000100:00100000:0.0:1789760717.946716:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707202333130 time 707253974486 state 0x0 ref 1 00000100:00100000:0.0:1789760717.946738:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707254234902 time 707253974486 state 0x1 ref 1 00000100:00100000:0.0:1789760717.946776:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707254234902 time 707277706380 state 0x0 ref 1 00000100:00100000:0.0:1789760717.946793:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484170752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760717.946841:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19472384, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:0.0:1789760717.946849:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.946865:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:0.0:1789760717.946870:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760717.946875:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19472384->19476479] (req 19472384->19476479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760717.946887:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19472384->19476479] (req 19472384->19476479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760717.969288:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19472384 -> 19476480 00010000:00010000:3.0:1789760717.969304:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19472384->19476479] (req 19472384->19476479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.969314:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19472384->19476479] (req 19472384->19476479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.969320:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19472384->19476479] (req 19472384->19476479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.969325:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760717.969330:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19472384->19476479] (req 19472384->19476479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.969339:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19472384->19476479] (req 19472384->19476479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.969359:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19472384->19476479] (req 19472384->19476479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.969372:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207bbb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19472384->19476479] (req 19472384->19476479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760717.969476:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484170752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 22686us (23130us total) trans 4294972072 rc 0/0 00000100:00100000:3.0:1789760717.969486:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4783 00000100:00100000:3.0:1789760717.969507:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707254234902 time 707277706380 state 0x2 ref 0 00000001:00080000:1.0:1789760717.970212:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972072 is committed 00000100:00100000:0.0:1789760717.972179:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760717.972251:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484170880 00000100:00100000:1.0:1789760717.972260:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484170880 00000100:00100000:1.0:1789760717.972310:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707254234902 time 707277706380 state 0x0 ref 1 00000100:00100000:1.0:1789760717.972321:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707277966796 time 707277706380 state 0x1 ref 1 00000100:00100000:1.0:1789760717.972339:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707277966796 time 707303271494 state 0x0 ref 1 00000100:00100000:1.0:1789760717.972353:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484170880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760717.972397:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19476480, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:1.0:1789760717.972406:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.972419:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:1.0:1789760717.972423:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760717.972428:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19476480->19480575] (req 19476480->19480575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760717.972441:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19476480->19480575] (req 19476480->19480575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760717.990198:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972073 is committed 00002000:00010000:3.0:1789760717.990371:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19476480 -> 19480576 00010000:00010000:3.0:1789760717.990386:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19476480->19480575] (req 19476480->19480575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.990398:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19476480->19480575] (req 19476480->19480575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.990407:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19476480->19480575] (req 19476480->19480575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.990414:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760717.990418:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19476480->19480575] (req 19476480->19480575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.990428:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19476480->19480575] (req 19476480->19480575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.990451:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19476480->19480575] (req 19476480->19480575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.990460:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207bc2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19476480->19480575] (req 19476480->19480575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760717.990594:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484170880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 18239us (18417us total) trans 4294972073 rc 0/0 00000100:00100000:3.0:1789760717.990604:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4784 00000100:00100000:3.0:1789760717.990614:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707277966796 time 707303271494 state 0x2 ref 0 00000100:00100000:0.0:1789760717.994790:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760717.994838:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484171008 00000100:00100000:3.0:1789760717.994847:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484171008 00000100:00100000:3.0:1789760717.994897:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707277966796 time 707303271494 state 0x0 ref 1 00000100:00100000:3.0:1789760717.994908:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707303531910 time 707303271494 state 0x1 ref 1 00000100:00100000:3.0:1789760717.994933:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707303531910 time 707325864557 state 0x0 ref 1 00000100:00100000:3.0:1789760717.994946:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760717.994993:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19480576, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:3.0:1789760717.995013:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.995026:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:3.0:1789760717.995030:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760717.995034:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19480576->19484671] (req 19480576->19484671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760717.995044:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19480576->19484671] (req 19480576->19484671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760718.030108:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972074 is committed 00002000:00010000:0.0:1789760718.030194:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19480576 -> 19484672 00010000:00010000:0.0:1789760718.030205:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19480576->19484671] (req 19480576->19484671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.030215:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19480576->19484671] (req 19480576->19484671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.030221:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19480576->19484671] (req 19480576->19484671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.030226:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.030229:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19480576->19484671] (req 19480576->19484671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.030236:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19480576->19484671] (req 19480576->19484671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.030250:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19480576->19484671] (req 19480576->19484671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.030256:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207bc9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19480576->19484671] (req 19480576->19484671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.030381:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35432us (35603us total) trans 4294972074 rc 0/0 00000100:00100000:0.0:1789760718.030391:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4785 00000100:00100000:0.0:1789760718.030401:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707303531910 time 707325864557 state 0x2 ref 0 00000100:00100000:0.0:1789760718.032455:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760718.032577:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484171136 00000100:00100000:3.0:1789760718.032585:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484171136 00000100:00100000:3.0:1789760718.032663:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707303531910 time 707325864557 state 0x0 ref 1 00000100:00100000:3.0:1789760718.032675:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707326124973 time 707325864557 state 0x1 ref 1 00000100:00100000:3.0:1789760718.032701:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707326124973 time 707363631621 state 0x0 ref 1 00000100:00100000:3.0:1789760718.032713:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484171136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760718.032755:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19484672, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:3.0:1789760718.032762:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.032772:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:3.0:1789760718.032775:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760718.032780:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19484672->19488767] (req 19484672->19488767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.032790:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19484672->19488767] (req 19484672->19488767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1789760718.033197:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:1.0:1789760718.071644:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19484672 -> 19488768 00010000:00010000:1.0:1789760718.071663:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19484672->19488767] (req 19484672->19488767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.071675:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19484672->19488767] (req 19484672->19488767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.071684:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19484672->19488767] (req 19484672->19488767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.071692:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760718.071696:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19484672->19488767] (req 19484672->19488767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.071705:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19484672->19488767] (req 19484672->19488767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.071726:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19484672->19488767] (req 19484672->19488767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.071734:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207bd0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19484672->19488767] (req 19484672->19488767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760718.071827:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484171136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39111us (39371us total) trans 4294972075 rc 0/0 00000100:00100000:1.0:1789760718.071837:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4786 00000100:00100000:1.0:1789760718.071846:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707326124973 time 707363631621 state 0x2 ref 0 00000001:00080000:3.0:1789760718.072342:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972075 is committed 00000100:00100000:1.0:1789760718.075445:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760718.075751:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484171264 00000100:00100000:0.0:1789760718.075760:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484171264 00000100:00100000:0.0:1789760718.075820:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707326124973 time 707363631621 state 0x0 ref 1 00000100:00100000:0.0:1789760718.075904:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707363892037 time 707363631621 state 0x1 ref 1 00000100:00100000:0.0:1789760718.075972:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707363892037 time 707406902453 state 0x0 ref 1 00000100:00100000:0.0:1789760718.075986:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484171264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760718.076033:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19488768, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:0.0:1789760718.076041:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.076058:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:0.0:1789760718.076095:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760718.076104:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19488768->19492863] (req 19488768->19492863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.076118:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19488768->19492863] (req 19488768->19492863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760718.093262:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19488768 -> 19492864 00010000:00010000:0.0:1789760718.093294:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19488768->19492863] (req 19488768->19492863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.093307:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19488768->19492863] (req 19488768->19492863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.093316:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19488768->19492863] (req 19488768->19492863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.093325:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.093330:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19488768->19492863] (req 19488768->19492863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.093340:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19488768->19492863] (req 19488768->19492863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.093359:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19488768->19492863] (req 19488768->19492863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.093367:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bd7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19488768->19492863] (req 19488768->19492863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.093455:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484171264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 17467us (18014us total) trans 4294972076 rc 0/0 00000100:00100000:0.0:1789760718.093464:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4787 00000100:00100000:0.0:1789760718.093474:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707363892037 time 707406902453 state 0x2 ref 0 00000001:00080000:3.0:1789760718.093553:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972076 is committed 00000100:00100000:0.0:1789760718.096055:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760718.096251:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484171392 00000100:00100000:1.0:1789760718.096261:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484171392 00000100:00100000:1.0:1789760718.096314:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707363892037 time 707406902453 state 0x0 ref 1 00000100:00100000:1.0:1789760718.096330:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707407162869 time 707406902453 state 0x1 ref 1 00000100:00100000:3.0:1789760718.096411:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707407162869 time 707427339123 state 0x0 ref 1 00000100:00100000:3.0:1789760718.096428:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760718.096526:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19492864, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:2.0:1789760718.096534:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.096547:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:2.0:1789760718.096550:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760718.096554:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19492864->19496959] (req 19492864->19496959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.096564:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19492864->19496959] (req 19492864->19496959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760718.116352:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19492864 -> 19496960 00000001:00080000:0.0:1789760718.116363:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972077 is committed 00010000:00010000:2.0:1789760718.116373:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19492864->19496959] (req 19492864->19496959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.116387:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19492864->19496959] (req 19492864->19496959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.116397:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19492864->19496959] (req 19492864->19496959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.116405:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760718.116418:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19492864->19496959] (req 19492864->19496959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.116428:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19492864->19496959] (req 19492864->19496959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.116452:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19492864->19496959] (req 19492864->19496959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.116459:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207bde lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19492864->19496959] (req 19492864->19496959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760718.116710:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 20281us (20659us total) trans 4294972077 rc 0/0 00000100:00100000:2.0:1789760718.116721:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4788 00000100:00100000:2.0:1789760718.116732:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707407162869 time 707427339123 state 0x2 ref 0 00000100:00100000:2.0:1789760718.119117:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760718.119171:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484171520 00000100:00100000:3.0:1789760718.119182:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484171520 00000100:00100000:3.0:1789760718.119241:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707407162869 time 707427339123 state 0x0 ref 1 00000100:00100000:3.0:1789760718.119259:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707427599539 time 707427339123 state 0x1 ref 1 00000100:00100000:3.0:1789760718.119353:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707427599539 time 707450282807 state 0x0 ref 1 00000100:00100000:3.0:1789760718.119367:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760718.119426:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19496960, mtime: 0x6aad94cd, atime: 0x0, ctime: 0x6aad94cd, blocks: 0x9807 00010000:00010000:3.0:1789760718.119436:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.119453:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:3.0:1789760718.119458:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760718.119464:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19496960->19501055] (req 19496960->19501055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.119479:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19496960->19501055] (req 19496960->19501055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760718.144476:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972078 is committed 00002000:00010000:0.0:1789760718.144519:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19496960 -> 19501056 00002000:00010000:0.0:1789760718.144525:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760717 -> 1789760718 00002000:00010000:0.0:1789760718.144529:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760717 -> 1789760718 00010000:00010000:0.0:1789760718.144543:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19496960->19501055] (req 19496960->19501055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.144556:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19496960->19501055] (req 19496960->19501055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.144565:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19496960->19501055] (req 19496960->19501055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.144572:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.144577:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19496960->19501055] (req 19496960->19501055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.144851:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19496960->19501055] (req 19496960->19501055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.144880:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19496960->19501055] (req 19496960->19501055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.144888:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207be5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19496960->19501055] (req 19496960->19501055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.144981:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25611us (25866us total) trans 4294972078 rc 0/0 00000100:00100000:0.0:1789760718.144992:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4789 00000100:00100000:0.0:1789760718.145002:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707427599539 time 707450282807 state 0x2 ref 0 00000100:00100000:2.0:1789760718.151025:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760718.151116:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484171648 00000100:00100000:1.0:1789760718.151125:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484171648 00000100:00100000:1.0:1789760718.151181:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707427599539 time 707450282807 state 0x0 ref 1 00000100:00100000:1.0:1789760718.151196:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707450543223 time 707450282807 state 0x1 ref 1 00000100:00100000:1.0:1789760718.151224:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707450543223 time 707482156105 state 0x0 ref 1 00000100:00100000:1.0:1789760718.151237:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760718.151283:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19501056, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:1.0:1789760718.151291:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.151307:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:1.0:1789760718.151311:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760718.151317:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19501056->19505151] (req 19501056->19505151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.151331:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19501056->19505151] (req 19501056->19505151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760718.181261:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972079 is committed 00002000:00010000:0.0:1789760718.181363:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19501056 -> 19505152 00010000:00010000:0.0:1789760718.181379:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19501056->19505151] (req 19501056->19505151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.181391:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19501056->19505151] (req 19501056->19505151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.181400:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19501056->19505151] (req 19501056->19505151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.181407:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.181412:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19501056->19505151] (req 19501056->19505151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.181426:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19501056->19505151] (req 19501056->19505151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.181447:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19501056->19505151] (req 19501056->19505151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.181456:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396207bec lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19501056->19505151] (req 19501056->19505151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.181807:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30565us (30785us total) trans 4294972079 rc 0/0 00000100:00100000:0.0:1789760718.181819:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4790 00000100:00100000:0.0:1789760718.181830:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707450543223 time 707482156105 state 0x2 ref 0 00000100:00100000:2.0:1789760718.186243:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760718.186275:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484171776 00000100:00100000:0.0:1789760718.186285:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484171776 00000100:00100000:0.0:1789760718.186350:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707450543223 time 707482156105 state 0x0 ref 1 00000100:00100000:0.0:1789760718.186367:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707482416521 time 707482156105 state 0x1 ref 1 00000100:00100000:0.0:1789760718.186446:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707482416521 time 707517376665 state 0x0 ref 1 00000100:00100000:0.0:1789760718.186463:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760718.186541:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19505152, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:0.0:1789760718.186551:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.186568:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:0.0:1789760718.186572:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760718.186579:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19505152->19509247] (req 19505152->19509247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.186595:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19505152->19509247] (req 19505152->19509247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760718.221617:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5082300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508910976:0@lo:41:osp-pre-1-0.0 00000100:00100000:2.0:1789760718.221685:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760718.225048:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508910976 00000100:00100000:3.0:1789760718.225061:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508910976 00000100:00100000:3.0:1789760718.225125:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 422 00000100:00100000:3.0:1789760718.225134:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508910976:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:3.0:1789760718.225171:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:3.0:1789760718.225248:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508910976:12345-0@lo:41:osp-pre-1-0.0 Request processed in 114us (3562us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760718.225255:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 422 00000100:00100000:3.0:1789760718.225320:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5082300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508910976:0@lo:41:osp-pre-1-0.0 00000001:00080000:1.0:1789760718.230331:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972080 is committed 00002000:00010000:2.0:1789760718.230487:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19505152 -> 19509248 00010000:00010000:2.0:1789760718.230516:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19505152->19509247] (req 19505152->19509247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.230545:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19505152->19509247] (req 19505152->19509247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.230558:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19505152->19509247] (req 19505152->19509247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.230566:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760718.230571:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19505152->19509247] (req 19505152->19509247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.230581:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19505152->19509247] (req 19505152->19509247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.230599:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19505152->19509247] (req 19505152->19509247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.230607:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207bf3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19505152->19509247] (req 19505152->19509247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760718.230832:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484171776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44367us (44593us total) trans 4294972080 rc 0/0 00000100:00100000:2.0:1789760718.230843:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4791 00000100:00100000:2.0:1789760718.230852:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707482416521 time 707517376665 state 0x2 ref 0 00000100:00100000:2.0:1789760718.234960:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760718.235144:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484171904 00000100:00100000:1.0:1789760718.235154:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484171904 00000100:00100000:1.0:1789760718.235206:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707482416521 time 707517376665 state 0x0 ref 1 00000100:00100000:1.0:1789760718.235222:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707517637081 time 707517376665 state 0x1 ref 1 00000100:00100000:1.0:1789760718.235283:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707517637081 time 707566213796 state 0x0 ref 1 00000100:00100000:1.0:1789760718.235296:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484171904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760718.235346:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19509248, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:1.0:1789760718.235356:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.235372:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:1.0:1789760718.235376:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760718.235381:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19509248->19513343] (req 19509248->19513343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.235394:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19509248->19513343] (req 19509248->19513343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760718.282187:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972081 is committed 00002000:00010000:0.0:1789760718.282811:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19509248 -> 19513344 00010000:00010000:0.0:1789760718.282826:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19509248->19513343] (req 19509248->19513343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.282840:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19509248->19513343] (req 19509248->19513343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.282849:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19509248->19513343] (req 19509248->19513343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.282856:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.282861:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19509248->19513343] (req 19509248->19513343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.282870:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19509248->19513343] (req 19509248->19513343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.282889:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19509248->19513343] (req 19509248->19513343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.282898:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396207bfa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19509248->19513343] (req 19509248->19513343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.282992:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484171904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47692us (48036us total) trans 4294972081 rc 0/0 00000100:00100000:0.0:1789760718.283001:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4792 00000100:00100000:0.0:1789760718.283011:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707517637081 time 707566213796 state 0x2 ref 0 00000100:00100000:1.0:1789760718.286851:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760718.286996:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484172032 00000100:00100000:3.0:1789760718.287007:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484172032 00000100:00100000:3.0:1789760718.287059:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707517637081 time 707566213796 state 0x0 ref 1 00000100:00100000:3.0:1789760718.287113:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707566474212 time 707566213796 state 0x1 ref 1 00000100:00100000:3.0:1789760718.287145:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707566474212 time 707618075485 state 0x0 ref 1 00000100:00100000:3.0:1789760718.287159:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484172032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760718.287208:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19513344, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:3.0:1789760718.287217:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.287234:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:3.0:1789760718.287240:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760718.287246:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19513344->19517439] (req 19513344->19517439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.287260:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19513344->19517439] (req 19513344->19517439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760718.340050:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972082 is committed 00002000:00010000:0.0:1789760718.340201:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19513344 -> 19517440 00010000:00010000:0.0:1789760718.340218:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19513344->19517439] (req 19513344->19517439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.340232:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19513344->19517439] (req 19513344->19517439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.340240:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19513344->19517439] (req 19513344->19517439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.340247:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.340252:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19513344->19517439] (req 19513344->19517439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.340260:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19513344->19517439] (req 19513344->19517439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.340354:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19513344->19517439] (req 19513344->19517439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.340362:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c01 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19513344->19517439] (req 19513344->19517439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.340455:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484172032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53295us (53608us total) trans 4294972082 rc 0/0 00000100:00100000:0.0:1789760718.340465:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4793 00000100:00100000:0.0:1789760718.340474:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707566474212 time 707618075485 state 0x2 ref 0 00000100:00100000:2.0:1789760718.344343:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760718.344435:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484172160 00000100:00100000:0.0:1789760718.344445:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484172160 00000100:00100000:0.0:1789760718.344512:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707566474212 time 707618075485 state 0x0 ref 1 00000100:00100000:0.0:1789760718.344528:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707618335901 time 707618075485 state 0x1 ref 1 00000100:00100000:0.0:1789760718.344603:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707618335901 time 707675533043 state 0x0 ref 1 00000100:00100000:0.0:1789760718.344618:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484172160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760718.344667:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19517440, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:0.0:1789760718.344675:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.344690:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:0.0:1789760718.344696:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760718.344702:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19517440->19521535] (req 19517440->19521535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.344715:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19517440->19521535] (req 19517440->19521535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760718.379015:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972083 is committed 00002000:00010000:0.0:1789760718.379125:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19517440 -> 19521536 00010000:00010000:0.0:1789760718.379140:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19517440->19521535] (req 19517440->19521535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.379152:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19517440->19521535] (req 19517440->19521535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.379162:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19517440->19521535] (req 19517440->19521535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.379170:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.379174:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19517440->19521535] (req 19517440->19521535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.379183:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19517440->19521535] (req 19517440->19521535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.379203:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19517440->19521535] (req 19517440->19521535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.379211:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207c08 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19517440->19521535] (req 19517440->19521535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.379353:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484172160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34735us (35082us total) trans 4294972083 rc 0/0 00000100:00100000:0.0:1789760718.379363:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4794 00000100:00100000:0.0:1789760718.379373:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707618335901 time 707675533043 state 0x2 ref 0 00000100:00100000:2.0:1789760718.383642:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760718.383725:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484172288 00000100:00100000:0.0:1789760718.383733:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484172288 00000100:00100000:0.0:1789760718.383787:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707618335901 time 707675533043 state 0x0 ref 1 00000100:00100000:0.0:1789760718.383801:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707675793459 time 707675533043 state 0x1 ref 1 00000100:00100000:0.0:1789760718.383860:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707675793459 time 707714791164 state 0x0 ref 1 00000100:00100000:0.0:1789760718.383873:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484172288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760718.383916:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19521536, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:0.0:1789760718.383924:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.383940:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:0.0:1789760718.383944:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760718.383950:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19521536->19525631] (req 19521536->19525631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.383963:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19521536->19525631] (req 19521536->19525631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760718.460610:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972084 is committed 00002000:00010000:3.0:1789760718.460837:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19521536 -> 19525632 00010000:00010000:3.0:1789760718.460853:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19521536->19525631] (req 19521536->19525631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.460867:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19521536->19525631] (req 19521536->19525631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.460877:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19521536->19525631] (req 19521536->19525631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.460884:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760718.460889:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19521536->19525631] (req 19521536->19525631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.460899:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19521536->19525631] (req 19521536->19525631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.460920:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19521536->19525631] (req 19521536->19525631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.460928:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207c0f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19521536->19525631] (req 19521536->19525631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760718.461022:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484172288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77146us (77383us total) trans 4294972084 rc 0/0 00000100:00100000:3.0:1789760718.461033:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4795 00000100:00100000:3.0:1789760718.461043:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707675793459 time 707714791164 state 0x2 ref 0 00000100:00100000:1.0:1789760718.465113:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760718.474312:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484172416 00000100:00100000:3.0:1789760718.474326:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484172416 00000100:00100000:3.0:1789760718.474382:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707675793459 time 707714791164 state 0x0 ref 1 00000100:00100000:3.0:1789760718.474399:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707715051580 time 707714791164 state 0x1 ref 1 00000100:00100000:3.0:1789760718.474421:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707715051580 time 707805352993 state 0x0 ref 1 00000100:00100000:3.0:1789760718.474433:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484172416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760718.474476:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19525632, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:3.0:1789760718.474484:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.474512:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:3.0:1789760718.474516:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760718.474743:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19525632->19529727] (req 19525632->19529727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.474759:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19525632->19529727] (req 19525632->19529727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760718.547850:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972085 is committed 00002000:00010000:3.0:1789760718.547992:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19525632 -> 19529728 00010000:00010000:3.0:1789760718.548010:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19525632->19529727] (req 19525632->19529727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.548022:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19525632->19529727] (req 19525632->19529727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.548031:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19525632->19529727] (req 19525632->19529727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.548038:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760718.548042:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19525632->19529727] (req 19525632->19529727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.548050:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19525632->19529727] (req 19525632->19529727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.548100:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19525632->19529727] (req 19525632->19529727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.548107:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c16 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19525632->19529727] (req 19525632->19529727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760718.548188:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484172416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73752us (83081us total) trans 4294972085 rc 0/0 00000100:00100000:3.0:1789760718.548201:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4796 00000100:00100000:3.0:1789760718.548210:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707715051580 time 707805352993 state 0x2 ref 0 00000100:00100000:0.0:1789760718.554384:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760718.554456:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484172544 00000100:00100000:3.0:1789760718.554466:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484172544 00000100:00100000:3.0:1789760718.554533:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707715051580 time 707805352993 state 0x0 ref 1 00000100:00100000:3.0:1789760718.554548:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707805613409 time 707805352993 state 0x1 ref 1 00000100:00100000:3.0:1789760718.554578:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707805613409 time 707885509265 state 0x0 ref 1 00000100:00100000:3.0:1789760718.554590:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484172544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760718.554635:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19529728, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:3.0:1789760718.554643:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.554657:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:3.0:1789760718.554661:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760718.554667:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19529728->19533823] (req 19529728->19533823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.554869:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19529728->19533823] (req 19529728->19533823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760718.619413:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19529728 -> 19533824 00010000:00010000:3.0:1789760718.619432:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19529728->19533823] (req 19529728->19533823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.619445:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19529728->19533823] (req 19529728->19533823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.619453:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19529728->19533823] (req 19529728->19533823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.619461:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760718.619465:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19529728->19533823] (req 19529728->19533823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.619475:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19529728->19533823] (req 19529728->19533823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.619505:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19529728->19533823] (req 19529728->19533823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.619513:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c1d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19529728->19533823] (req 19529728->19533823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760718.619636:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484172544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65043us (65256us total) trans 4294972086 rc 0/0 00000100:00100000:3.0:1789760718.619647:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4797 00000100:00100000:3.0:1789760718.619656:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707805613409 time 707885509265 state 0x2 ref 0 00000001:00080000:0.0:1789760718.620669:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972086 is committed 00000100:00100000:0.0:1789760718.623199:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760718.623457:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484172672 00000100:00100000:1.0:1789760718.623467:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484172672 00000100:00100000:1.0:1789760718.623527:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707805613409 time 707885509265 state 0x0 ref 1 00000100:00100000:1.0:1789760718.623542:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707885769681 time 707885509265 state 0x1 ref 1 00000100:00100000:1.0:1789760718.623659:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707885769681 time 707954588441 state 0x0 ref 1 00000100:00100000:1.0:1789760718.623673:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484172672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760718.623800:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19533824, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:1.0:1789760718.623810:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.623826:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:1.0:1789760718.623830:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760718.623836:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19533824->19537919] (req 19533824->19537919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.623850:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19533824->19537919] (req 19533824->19537919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760718.656546:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972087 is committed 00002000:00010000:3.0:1789760718.656594:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19533824 -> 19537920 00010000:00010000:3.0:1789760718.656610:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19533824->19537919] (req 19533824->19537919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.656623:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19533824->19537919] (req 19533824->19537919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.656631:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19533824->19537919] (req 19533824->19537919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.656639:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760718.656644:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19533824->19537919] (req 19533824->19537919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.656654:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19533824->19537919] (req 19533824->19537919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.656671:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19533824->19537919] (req 19533824->19537919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.657850:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207c24 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19533824->19537919] (req 19533824->19537919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760718.657957:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484172672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34281us (34762us total) trans 4294972087 rc 0/0 00000100:00100000:3.0:1789760718.657968:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4798 00000100:00100000:3.0:1789760718.657978:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707885769681 time 707954588441 state 0x2 ref 0 00000100:00100000:1.0:1789760718.662896:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760718.663111:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484172800 00000100:00100000:0.0:1789760718.663123:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484172800 00000100:00100000:0.0:1789760718.663183:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707885769681 time 707954588441 state 0x0 ref 1 00000100:00100000:0.0:1789760718.663199:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707954848857 time 707954588441 state 0x1 ref 1 00000100:00100000:0.0:1789760718.663230:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707954848857 time 707994160654 state 0x0 ref 1 00000100:00100000:0.0:1789760718.663245:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484172800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760718.663403:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19537920, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:0.0:1789760718.663412:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.663428:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=707s v=(0 0) 00010000:00010000:0.0:1789760718.663433:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760718.663439:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19537920->19542015] (req 19537920->19542015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.663453:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19537920->19542015] (req 19537920->19542015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760718.724760:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972088 is committed 00002000:00010000:3.0:1789760718.724892:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19537920 -> 19542016 00010000:00010000:3.0:1789760718.724913:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19537920->19542015] (req 19537920->19542015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.724929:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19537920->19542015] (req 19537920->19542015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.724943:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19537920->19542015] (req 19537920->19542015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.724951:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760718.724956:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19537920->19542015] (req 19537920->19542015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.724965:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19537920->19542015] (req 19537920->19542015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.724994:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19537920->19542015] (req 19537920->19542015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.725002:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207c2b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19537920->19542015] (req 19537920->19542015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760718.725188:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484172800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61933us (62288us total) trans 4294972088 rc 0/0 00000100:00100000:3.0:1789760718.725201:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4799 00000100:00100000:3.0:1789760718.725215:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707954848857 time 707994160654 state 0x2 ref 0 00000100:00100000:0.0:1789760718.730984:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760718.731113:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484172928 00000100:00100000:3.0:1789760718.731126:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484172928 00000100:00100000:3.0:1789760718.731201:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707954848857 time 707994160654 state 0x0 ref 1 00000100:00100000:3.0:1789760718.731219:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707994421070 time 707994160654 state 0x1 ref 1 00000100:00100000:3.0:1789760718.731255:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707994421070 time 708062186687 state 0x0 ref 1 00000100:00100000:3.0:1789760718.731279:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484172928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760718.731346:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19542016, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:3.0:1789760718.731357:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.731379:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:3.0:1789760718.731384:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760718.731391:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19542016->19546111] (req 19542016->19546111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.731409:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19542016->19546111] (req 19542016->19546111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760718.732884:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:3.0:1789760718.762093:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19542016 -> 19546112 00010000:00010000:3.0:1789760718.762124:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19542016->19546111] (req 19542016->19546111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.762147:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19542016->19546111] (req 19542016->19546111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.762158:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19542016->19546111] (req 19542016->19546111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.762165:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760718.762171:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19542016->19546111] (req 19542016->19546111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.762180:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19542016->19546111] (req 19542016->19546111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.762209:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19542016->19546111] (req 19542016->19546111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.762218:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c32 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19542016->19546111] (req 19542016->19546111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760718.762230:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972089 is committed 00000100:00100000:3.0:1789760718.762347:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484172928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31071us (31370us total) trans 4294972089 rc 0/0 00000100:00100000:3.0:1789760718.762360:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4800 00000100:00100000:3.0:1789760718.762373:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707994421070 time 708062186687 state 0x2 ref 0 00000100:00100000:0.0:1789760718.765522:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760718.765752:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484173056 00000100:00100000:3.0:1789760718.765766:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484173056 00000100:00100000:3.0:1789760718.765841:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 707994421070 time 708062186687 state 0x0 ref 1 00000100:00100000:3.0:1789760718.765862:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708062447103 time 708062186687 state 0x1 ref 1 00000100:00100000:3.0:1789760718.765946:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708062447103 time 708096874822 state 0x0 ref 1 00000100:00100000:3.0:1789760718.765974:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760718.766040:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19546112, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:3.0:1789760718.766052:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.766121:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:3.0:1789760718.766127:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760718.766136:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19546112->19550207] (req 19546112->19550207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.766153:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19546112->19550207] (req 19546112->19550207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760718.798366:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972090 is committed 00002000:00010000:0.0:1789760718.798506:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19546112 -> 19550208 00010000:00010000:0.0:1789760718.798526:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19546112->19550207] (req 19546112->19550207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.798542:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19546112->19550207] (req 19546112->19550207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.798551:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19546112->19550207] (req 19546112->19550207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.798559:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.798564:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19546112->19550207] (req 19546112->19550207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.798574:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19546112->19550207] (req 19546112->19550207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.798726:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19546112->19550207] (req 19546112->19550207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.798737:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207c39 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19546112->19550207] (req 19546112->19550207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.798875:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32899us (33365us total) trans 4294972090 rc 0/0 00000100:00100000:0.0:1789760718.798887:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4801 00000100:00100000:0.0:1789760718.798899:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708062447103 time 708096874822 state 0x2 ref 0 00000100:00100000:1.0:1789760718.801579:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760718.801656:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484173184 00000100:00100000:0.0:1789760718.801667:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484173184 00000100:00100000:0.0:1789760718.801746:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708062447103 time 708096874822 state 0x0 ref 1 00000100:00100000:0.0:1789760718.801766:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708097135238 time 708096874822 state 0x1 ref 1 00000100:00100000:0.0:1789760718.801799:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708097135238 time 708132729543 state 0x0 ref 1 00000100:00100000:0.0:1789760718.801811:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760718.801864:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19550208, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:0.0:1789760718.801872:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.801889:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:0.0:1789760718.801893:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760718.801898:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19550208->19554303] (req 19550208->19554303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.801911:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19550208->19554303] (req 19550208->19554303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760718.821552:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972091 is committed 00002000:00010000:0.0:1789760718.821601:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19550208 -> 19554304 00010000:00010000:0.0:1789760718.821622:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19550208->19554303] (req 19550208->19554303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.821637:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19550208->19554303] (req 19550208->19554303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.821647:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19550208->19554303] (req 19550208->19554303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.821656:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.821661:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19550208->19554303] (req 19550208->19554303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.821671:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19550208->19554303] (req 19550208->19554303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.821695:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19550208->19554303] (req 19550208->19554303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.821705:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c40 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19550208->19554303] (req 19550208->19554303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.821818:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 20002us (20243us total) trans 4294972091 rc 0/0 00000100:00100000:0.0:1789760718.821834:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4802 00000100:00100000:0.0:1789760718.821846:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708097135238 time 708132729543 state 0x2 ref 0 00000100:00100000:2.0:1789760718.823876:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760718.824035:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484173312 00000100:00100000:0.0:1789760718.824047:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484173312 00000100:00100000:0.0:1789760718.824168:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708097135238 time 708132729543 state 0x0 ref 1 00000100:00100000:0.0:1789760718.824190:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708132989959 time 708132729543 state 0x1 ref 1 00000100:00100000:1.0:1789760718.824313:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708132989959 time 708155239986 state 0x0 ref 1 00000100:00100000:1.0:1789760718.824331:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760718.824396:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19554304, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:1.0:1789760718.824407:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.824428:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:1.0:1789760718.824433:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760718.824441:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19554304->19558399] (req 19554304->19558399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760718.824458:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19554304->19558399] (req 19554304->19558399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760718.845744:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972092 is committed 00002000:00010000:0.0:1789760718.845985:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19554304 -> 19558400 00010000:00010000:0.0:1789760718.846007:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19554304->19558399] (req 19554304->19558399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.846022:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19554304->19558399] (req 19554304->19558399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.846032:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19554304->19558399] (req 19554304->19558399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.846041:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.846047:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19554304->19558399] (req 19554304->19558399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.846057:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19554304->19558399] (req 19554304->19558399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.846314:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19554304->19558399] (req 19554304->19558399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.846324:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207c47 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19554304->19558399] (req 19554304->19558399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760718.846708:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 22373us (22836us total) trans 4294972092 rc 0/0 00000100:00100000:1.0:1789760718.846722:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4803 00000100:00100000:1.0:1789760718.846735:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708132989959 time 708155239986 state 0x2 ref 0 00000100:00100000:2.0:1789760718.851338:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760718.857904:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484173440 00000100:00100000:3.0:1789760718.857928:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484173440 00000100:00100000:3.0:1789760718.858009:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708132989959 time 708155239986 state 0x0 ref 1 00000100:00100000:3.0:1789760718.858029:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708155500402 time 708155239986 state 0x1 ref 1 00000100:00100000:3.0:1789760718.858414:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708155500402 time 708188996919 state 0x0 ref 1 00000100:00100000:3.0:1789760718.858434:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484173440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760718.858510:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19558400, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:3.0:1789760718.858522:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.858542:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:3.0:1789760718.858547:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760718.858554:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19558400->19562495] (req 19558400->19562495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.858570:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19558400->19562495] (req 19558400->19562495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760718.919447:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972093 is committed 00002000:00010000:0.0:1789760718.919598:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19558400 -> 19562496 00010000:00010000:0.0:1789760718.919619:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19558400->19562495] (req 19558400->19562495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.919635:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19558400->19562495] (req 19558400->19562495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.919645:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19558400->19562495] (req 19558400->19562495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.919654:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.919660:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19558400->19562495] (req 19558400->19562495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.919672:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19558400->19562495] (req 19558400->19562495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.919699:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19558400->19562495] (req 19558400->19562495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.919707:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207c4e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19558400->19562495] (req 19558400->19562495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.919944:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484173440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61507us (68613us total) trans 4294972093 rc 0/0 00000100:00100000:0.0:1789760718.919957:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4804 00000100:00100000:0.0:1789760718.919982:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708155500402 time 708188996919 state 0x2 ref 0 00000100:00100000:2.0:1789760718.933119:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760718.933229:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484173568 00000100:00100000:3.0:1789760718.933243:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484173568 00000100:00100000:3.0:1789760718.933343:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708155500402 time 708188996919 state 0x0 ref 1 00000100:00100000:3.0:1789760718.933361:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708189257335 time 708188996919 state 0x1 ref 1 00000100:00100000:3.0:1789760718.933396:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708189257335 time 708264327211 state 0x0 ref 1 00000100:00100000:3.0:1789760718.933411:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760718.933478:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19562496, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:3.0:1789760718.933509:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.933537:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:3.0:1789760718.933542:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760718.933550:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19562496->19566591] (req 19562496->19566591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760718.933568:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19562496->19566591] (req 19562496->19566591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760718.985910:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972094 is committed 00002000:00010000:0.0:1789760718.986023:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19562496 -> 19566592 00010000:00010000:0.0:1789760718.986325:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19562496->19566591] (req 19562496->19566591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.986342:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19562496->19566591] (req 19562496->19566591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.986352:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19562496->19566591] (req 19562496->19566591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.986361:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760718.986367:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19562496->19566591] (req 19562496->19566591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.986377:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19562496->19566591] (req 19562496->19566591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.986403:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19562496->19566591] (req 19562496->19566591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.986417:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207c55 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19562496->19566591] (req 19562496->19566591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760718.986755:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53338us (53686us total) trans 4294972094 rc 0/0 00000100:00100000:0.0:1789760718.986769:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4805 00000100:00100000:0.0:1789760718.986785:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708189257335 time 708264327211 state 0x2 ref 0 00000100:00100000:2.0:1789760718.990185:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760718.990236:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484173696 00000100:00100000:0.0:1789760718.990250:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484173696 00000100:00100000:0.0:1789760718.990340:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708189257335 time 708264327211 state 0x0 ref 1 00000100:00100000:0.0:1789760718.990359:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708264587627 time 708264327211 state 0x1 ref 1 00000100:00100000:0.0:1789760718.990402:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708264587627 time 708321332195 state 0x0 ref 1 00000100:00100000:0.0:1789760718.990430:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484173696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760718.990514:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19566592, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:0.0:1789760718.990524:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.990545:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:0.0:1789760718.990551:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760718.990558:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19566592->19570687] (req 19566592->19570687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760718.990580:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19566592->19570687] (req 19566592->19570687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760719.015013:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972095 is committed 00002000:00010000:2.0:1789760719.015587:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19566592 -> 19570688 00010000:00010000:2.0:1789760719.015610:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19566592->19570687] (req 19566592->19570687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.015629:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19566592->19570687] (req 19566592->19570687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.015639:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19566592->19570687] (req 19566592->19570687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.015646:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760719.015651:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19566592->19570687] (req 19566592->19570687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.015661:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19566592->19570687] (req 19566592->19570687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.015686:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19566592->19570687] (req 19566592->19570687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.015694:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207c5c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19566592->19570687] (req 19566592->19570687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760719.015819:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484173696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25398us (25642us total) trans 4294972095 rc 0/0 00000100:00100000:2.0:1789760719.015829:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4806 00000100:00100000:2.0:1789760719.015840:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708264587627 time 708321332195 state 0x2 ref 0 00000100:00100000:2.0:1789760719.018521:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760719.018966:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484173824 00000100:00100000:1.0:1789760719.018980:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484173824 00000100:00100000:1.0:1789760719.019051:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708264587627 time 708321332195 state 0x0 ref 1 00000100:00100000:1.0:1789760719.019154:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708321592611 time 708321332195 state 0x1 ref 1 00000100:00100000:1.0:1789760719.019187:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708321592611 time 708350117919 state 0x0 ref 1 00000100:00100000:1.0:1789760719.019203:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760719.019331:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19570688, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:1.0:1789760719.019343:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.019365:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:1.0:1789760719.019370:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760719.019377:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19570688->19574783] (req 19570688->19574783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.019393:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19570688->19574783] (req 19570688->19574783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760719.042966:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972096 is committed 00002000:00010000:0.0:1789760719.043141:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19570688 -> 19574784 00010000:00010000:0.0:1789760719.043161:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19570688->19574783] (req 19570688->19574783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.043175:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19570688->19574783] (req 19570688->19574783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.043186:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19570688->19574783] (req 19570688->19574783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.043194:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760719.043199:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19570688->19574783] (req 19570688->19574783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.043208:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19570688->19574783] (req 19570688->19574783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.043232:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19570688->19574783] (req 19570688->19574783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.043240:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207c63 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19570688->19574783] (req 19570688->19574783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.043362:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484173824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24155us (24847us total) trans 4294972096 rc 0/0 00000100:00100000:0.0:1789760719.043375:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4807 00000100:00100000:0.0:1789760719.043387:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708321592611 time 708350117919 state 0x2 ref 0 00000100:00100000:1.0:1789760719.046338:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760719.046459:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484173952 00000100:00100000:3.0:1789760719.046527:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484173952 00000100:00100000:3.0:1789760719.046656:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708321592611 time 708350117919 state 0x0 ref 1 00000100:00100000:3.0:1789760719.046676:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708350378335 time 708350117919 state 0x1 ref 1 00000100:00100000:3.0:1789760719.046749:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708350378335 time 708377679861 state 0x0 ref 1 00000100:00100000:3.0:1789760719.046760:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484173952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760719.046819:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19574784, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:3.0:1789760719.046829:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.046849:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:3.0:1789760719.046853:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760719.046860:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19574784->19578879] (req 19574784->19578879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.046877:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19574784->19578879] (req 19574784->19578879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760719.094435:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19574784 -> 19578880 00010000:00010000:0.0:1789760719.094464:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19574784->19578879] (req 19574784->19578879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.094480:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19574784->19578879] (req 19574784->19578879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.094500:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19574784->19578879] (req 19574784->19578879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.094507:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00000001:00080000:3.0:1789760719.094508:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972097 is committed 00010000:00010000:0.0:1789760719.094513:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19574784->19578879] (req 19574784->19578879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.094522:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19574784->19578879] (req 19574784->19578879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.094547:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19574784->19578879] (req 19574784->19578879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.094562:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207c6a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19574784->19578879] (req 19574784->19578879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.094787:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484173952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48020us (48453us total) trans 4294972097 rc 0/0 00000100:00100000:0.0:1789760719.094800:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4808 00000100:00100000:0.0:1789760719.094812:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708350378335 time 708377679861 state 0x2 ref 0 00000100:00100000:2.0:1789760719.099905:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760719.100010:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484174080 00000100:00100000:0.0:1789760719.100025:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484174080 00000100:00100000:0.0:1789760719.100168:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708350378335 time 708377679861 state 0x0 ref 1 00000100:00100000:0.0:1789760719.100187:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708377940277 time 708377679861 state 0x1 ref 1 00000100:00100000:0.0:1789760719.100212:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708377940277 time 708431143535 state 0x0 ref 1 00000100:00100000:0.0:1789760719.100225:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484174080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760719.100290:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19578880, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:0.0:1789760719.100300:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.100322:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:0.0:1789760719.100327:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760719.100335:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19578880->19582975] (req 19578880->19582975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.100357:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19578880->19582975] (req 19578880->19582975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760719.121030:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972098 is committed 00002000:00010000:0.0:1789760719.121140:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19578880 -> 19582976 00010000:00010000:0.0:1789760719.121159:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19578880->19582975] (req 19578880->19582975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.121176:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19578880->19582975] (req 19578880->19582975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.121186:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19578880->19582975] (req 19578880->19582975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.121194:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760719.121199:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19578880->19582975] (req 19578880->19582975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.121206:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19578880->19582975] (req 19578880->19582975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.121227:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19578880->19582975] (req 19578880->19582975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.121231:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c71 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19578880->19582975] (req 19578880->19582975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.121350:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484174080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 21120us (21453us total) trans 4294972098 rc 0/0 00000100:00100000:0.0:1789760719.121363:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4809 00000100:00100000:0.0:1789760719.121375:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708377940277 time 708431143535 state 0x2 ref 0 00000100:00100000:2.0:1789760719.132191:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760719.132415:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484174208 00000100:00100000:0.0:1789760719.132428:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484174208 00000100:00100000:0.0:1789760719.132575:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708377940277 time 708431143535 state 0x0 ref 1 00000100:00100000:0.0:1789760719.132594:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708431403951 time 708431143535 state 0x1 ref 1 00000100:00100000:0.0:1789760719.132631:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708431403951 time 708463561573 state 0x0 ref 1 00000100:00100000:0.0:1789760719.132648:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484174208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760719.132716:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19582976, mtime: 0x6aad94ce, atime: 0x0, ctime: 0x6aad94ce, blocks: 0x9807 00010000:00010000:0.0:1789760719.132728:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.132757:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:0.0:1789760719.132763:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760719.132771:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19582976->19587071] (req 19582976->19587071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.132787:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19582976->19587071] (req 19582976->19587071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760719.159359:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972099 is committed 00002000:00010000:0.0:1789760719.159448:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19582976 -> 19587072 00002000:00010000:0.0:1789760719.159454:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760718 -> 1789760719 00002000:00010000:0.0:1789760719.159456:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760718 -> 1789760719 00010000:00010000:0.0:1789760719.159473:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19582976->19587071] (req 19582976->19587071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.159499:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19582976->19587071] (req 19582976->19587071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.159508:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19582976->19587071] (req 19582976->19587071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.159516:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760719.159521:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19582976->19587071] (req 19582976->19587071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.159531:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19582976->19587071] (req 19582976->19587071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.159564:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19582976->19587071] (req 19582976->19587071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.159573:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c78 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19582976->19587071] (req 19582976->19587071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.159699:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484174208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27049us (27517us total) trans 4294972099 rc 0/0 00000100:00100000:0.0:1789760719.159712:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4810 00000100:00100000:0.0:1789760719.159723:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708431403951 time 708463561573 state 0x2 ref 0 00000100:00100000:1.0:1789760719.166706:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760719.166774:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484174336 00000100:00100000:0.0:1789760719.166787:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484174336 00000100:00100000:0.0:1789760719.166863:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708431403951 time 708463561573 state 0x0 ref 1 00000100:00100000:0.0:1789760719.166878:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708463821989 time 708463561573 state 0x1 ref 1 00000100:00100000:0.0:1789760719.166946:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708463821989 time 708497875869 state 0x0 ref 1 00000100:00100000:0.0:1789760719.166962:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484174336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760719.167036:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19587072, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:0.0:1789760719.167047:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.167094:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:0.0:1789760719.167100:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760719.167108:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19587072->19591167] (req 19587072->19591167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.167124:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19587072->19591167] (req 19587072->19591167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760719.200873:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972100 is committed 00002000:00010000:1.0:1789760719.201337:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19587072 -> 19591168 00010000:00010000:1.0:1789760719.201376:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19587072->19591167] (req 19587072->19591167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.201392:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19587072->19591167] (req 19587072->19591167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.201407:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19587072->19591167] (req 19587072->19591167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.201414:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760719.201420:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19587072->19591167] (req 19587072->19591167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.201429:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19587072->19591167] (req 19587072->19591167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.201452:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19587072->19591167] (req 19587072->19591167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.201460:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c7f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19587072->19591167] (req 19587072->19591167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760719.201601:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484174336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34636us (34899us total) trans 4294972100 rc 0/0 00000100:00100000:1.0:1789760719.201612:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4811 00000100:00100000:1.0:1789760719.201624:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708463821989 time 708497875869 state 0x2 ref 0 00000100:00100000:2.0:1789760719.205772:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760719.205880:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484174464 00000100:00100000:3.0:1789760719.205894:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484174464 00000100:00100000:3.0:1789760719.205967:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708463821989 time 708497875869 state 0x0 ref 1 00000100:00100000:3.0:1789760719.205982:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708498136285 time 708497875869 state 0x1 ref 1 00000100:00100000:3.0:1789760719.206011:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708498136285 time 708536943392 state 0x0 ref 1 00000100:00100000:3.0:1789760719.206023:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484174464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760719.206119:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19591168, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:3.0:1789760719.206130:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.206150:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:3.0:1789760719.206155:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760719.206162:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19591168->19595263] (req 19591168->19595263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.206179:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19591168->19595263] (req 19591168->19595263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760719.231746:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972101 is committed 00002000:00010000:1.0:1789760719.231857:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19591168 -> 19595264 00010000:00010000:1.0:1789760719.231877:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19591168->19595263] (req 19591168->19595263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.231898:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19591168->19595263] (req 19591168->19595263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.231908:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19591168->19595263] (req 19591168->19595263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.231917:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760719.231923:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19591168->19595263] (req 19591168->19595263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.231933:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19591168->19595263] (req 19591168->19595263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.231959:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19591168->19595263] (req 19591168->19595263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.231967:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207c86 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19591168->19595263] (req 19591168->19595263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760719.232515:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484174464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26485us (26751us total) trans 4294972101 rc 0/0 00000100:00100000:1.0:1789760719.232529:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4812 00000100:00100000:1.0:1789760719.232543:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708498136285 time 708536943392 state 0x2 ref 0 00000100:00100000:2.0:1789760719.236740:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760719.236936:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484174592 00000100:00100000:3.0:1789760719.236949:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484174592 00000100:00100000:3.0:1789760719.237023:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708498136285 time 708536943392 state 0x0 ref 1 00000100:00100000:3.0:1789760719.237041:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708537203808 time 708536943392 state 0x1 ref 1 00000100:00100000:3.0:1789760719.237197:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708537203808 time 708568127150 state 0x0 ref 1 00000100:00100000:3.0:1789760719.237208:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7072850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484174592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760719.237276:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19595264, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:3.0:1789760719.237288:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.237310:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:3.0:1789760719.237314:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760719.237320:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19595264->19599359] (req 19595264->19599359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.237334:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19595264->19599359] (req 19595264->19599359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760719.276919:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972102 is committed 00002000:00010000:1.0:1789760719.277210:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19595264 -> 19599360 00010000:00010000:1.0:1789760719.277233:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19595264->19599359] (req 19595264->19599359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.277251:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19595264->19599359] (req 19595264->19599359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.277262:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19595264->19599359] (req 19595264->19599359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.277271:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760719.277277:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19595264->19599359] (req 19595264->19599359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.277289:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19595264->19599359] (req 19595264->19599359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.277315:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19595264->19599359] (req 19595264->19599359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.277324:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207c8d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19595264->19599359] (req 19595264->19599359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760719.277651:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7072850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484174592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40434us (40921us total) trans 4294972102 rc 0/0 00000100:00100000:1.0:1789760719.277663:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4813 00000100:00100000:1.0:1789760719.277675:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708537203808 time 708568127150 state 0x2 ref 0 00000100:00100000:2.0:1789760719.282009:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760719.284223:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484174720 00000100:00100000:0.0:1789760719.284245:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484174720 00000100:00100000:0.0:1789760719.284344:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708537203808 time 708568127150 state 0x0 ref 1 00000100:00100000:0.0:1789760719.284365:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708568387566 time 708568127150 state 0x1 ref 1 00000100:00100000:0.0:1789760719.284443:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708568387566 time 708615372620 state 0x0 ref 1 00000100:00100000:0.0:1789760719.284461:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484174720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760719.284557:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19599360, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:0.0:1789760719.284567:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.284586:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:0.0:1789760719.284591:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760719.284599:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19599360->19603455] (req 19599360->19603455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.284616:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19599360->19603455] (req 19599360->19603455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760719.336405:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972103 is committed 00002000:00010000:0.0:1789760719.336543:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19599360 -> 19603456 00010000:00010000:0.0:1789760719.336562:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19599360->19603455] (req 19599360->19603455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.336585:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19599360->19603455] (req 19599360->19603455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.336596:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19599360->19603455] (req 19599360->19603455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.336604:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760719.336610:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19599360->19603455] (req 19599360->19603455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.336620:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19599360->19603455] (req 19599360->19603455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.336646:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19599360->19603455] (req 19599360->19603455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.336655:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207c94 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19599360->19603455] (req 19599360->19603455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.336782:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484174720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52318us (54964us total) trans 4294972103 rc 0/0 00000100:00100000:0.0:1789760719.336794:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4814 00000100:00100000:0.0:1789760719.336808:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708568387566 time 708615372620 state 0x2 ref 0 00000100:00100000:3.0:1789760719.340507:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760719.340638:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484174848 00000100:00100000:1.0:1789760719.340651:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484174848 00000100:00100000:1.0:1789760719.340722:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708568387566 time 708615372620 state 0x0 ref 1 00000100:00100000:1.0:1789760719.340742:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708615633036 time 708615372620 state 0x1 ref 1 00000100:00100000:1.0:1789760719.340776:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708615633036 time 708671707118 state 0x0 ref 1 00000100:00100000:1.0:1789760719.340791:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484174848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760719.340859:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19603456, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:1.0:1789760719.340870:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.340964:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:1.0:1789760719.340970:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760719.340978:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19603456->19607551] (req 19603456->19607551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.340995:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19603456->19607551] (req 19603456->19607551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760719.399998:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972104 is committed 00002000:00010000:0.0:1789760719.400450:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19603456 -> 19607552 00010000:00010000:0.0:1789760719.400470:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19603456->19607551] (req 19603456->19607551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.400493:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19603456->19607551] (req 19603456->19607551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.400504:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19603456->19607551] (req 19603456->19607551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.400511:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760719.400517:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19603456->19607551] (req 19603456->19607551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.400526:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19603456->19607551] (req 19603456->19607551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.400552:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19603456->19607551] (req 19603456->19607551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.400560:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396207c9b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19603456->19607551] (req 19603456->19607551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.400781:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484174848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59985us (60291us total) trans 4294972104 rc 0/0 00000100:00100000:0.0:1789760719.400793:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4815 00000100:00100000:0.0:1789760719.400805:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708615633036 time 708671707118 state 0x2 ref 0 00000100:00100000:3.0:1789760719.405532:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760719.405643:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484174976 00000100:00100000:1.0:1789760719.405657:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484174976 00000100:00100000:1.0:1789760719.405733:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708615633036 time 708671707118 state 0x0 ref 1 00000100:00100000:1.0:1789760719.405753:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708671967534 time 708671707118 state 0x1 ref 1 00000100:00100000:1.0:1789760719.405791:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708671967534 time 708736721995 state 0x0 ref 1 00000100:00100000:1.0:1789760719.405806:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484174976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760719.405887:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19607552, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:1.0:1789760719.405898:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.405919:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:1.0:1789760719.405924:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760719.405932:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19607552->19611647] (req 19607552->19611647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.405951:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19607552->19611647] (req 19607552->19611647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760719.452822:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972105 is committed 00002000:00010000:3.0:1789760719.453270:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19607552 -> 19611648 00010000:00010000:3.0:1789760719.453291:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19607552->19611647] (req 19607552->19611647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.453307:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19607552->19611647] (req 19607552->19611647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.453317:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19607552->19611647] (req 19607552->19611647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.453325:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760719.453330:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19607552->19611647] (req 19607552->19611647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.453340:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19607552->19611647] (req 19607552->19611647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.453366:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19607552->19611647] (req 19607552->19611647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.453373:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ca2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19607552->19611647] (req 19607552->19611647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760719.453527:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484174976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47716us (48003us total) trans 4294972105 rc 0/0 00000100:00100000:3.0:1789760719.453541:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4816 00000100:00100000:3.0:1789760719.453553:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708671967534 time 708736721995 state 0x2 ref 0 00000100:00100000:3.0:1789760719.458046:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760719.458187:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484175104 00000100:00100000:1.0:1789760719.458202:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484175104 00000100:00100000:1.0:1789760719.458287:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708671967534 time 708736721995 state 0x0 ref 1 00000100:00100000:1.0:1789760719.458305:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708736982411 time 708736721995 state 0x1 ref 1 00000100:00100000:1.0:1789760719.458332:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708736982411 time 708789263114 state 0x0 ref 1 00000100:00100000:1.0:1789760719.458347:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484175104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760719.458817:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19611648, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:1.0:1789760719.458830:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.458852:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:1.0:1789760719.458857:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760719.458865:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19611648->19615743] (req 19611648->19615743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.459029:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19611648->19615743] (req 19611648->19615743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760719.516760:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972106 is committed 00002000:00010000:1.0:1789760719.516966:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19611648 -> 19615744 00010000:00010000:1.0:1789760719.516988:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19611648->19615743] (req 19611648->19615743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.517003:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19611648->19615743] (req 19611648->19615743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.517013:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19611648->19615743] (req 19611648->19615743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.517021:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760719.517027:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19611648->19615743] (req 19611648->19615743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.517037:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19611648->19615743] (req 19611648->19615743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.517089:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19611648->19615743] (req 19611648->19615743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.517099:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396207ca9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19611648->19615743] (req 19611648->19615743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760719.517276:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484175104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58917us (59230us total) trans 4294972106 rc 0/0 00000100:00100000:1.0:1789760719.517288:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4817 00000100:00100000:1.0:1789760719.517300:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708736982411 time 708789263114 state 0x2 ref 0 00000100:00100000:0.0:1789760719.521950:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760719.522234:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484175232 00000100:00100000:2.0:1789760719.522247:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484175232 00000100:00100000:2.0:1789760719.522451:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708736982411 time 708789263114 state 0x0 ref 1 00000100:00100000:2.0:1789760719.522473:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708789523530 time 708789263114 state 0x1 ref 1 00000100:00100000:2.0:1789760719.522669:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708789523530 time 708853597607 state 0x0 ref 1 00000100:00100000:2.0:1789760719.522688:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484175232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760719.522748:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19615744, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:2.0:1789760719.522759:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.522779:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:2.0:1789760719.522784:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760719.522791:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19615744->19619839] (req 19615744->19619839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.522807:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19615744->19619839] (req 19615744->19619839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760719.575753:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972107 is committed 00002000:00010000:3.0:1789760719.575940:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19615744 -> 19619840 00010000:00010000:3.0:1789760719.575961:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19615744->19619839] (req 19615744->19619839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.575977:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19615744->19619839] (req 19615744->19619839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.575988:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19615744->19619839] (req 19615744->19619839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.575996:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760719.576002:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19615744->19619839] (req 19615744->19619839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.576012:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19615744->19619839] (req 19615744->19619839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.576117:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19615744->19619839] (req 19615744->19619839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.576128:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396207cb0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19615744->19619839] (req 19615744->19619839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760719.576317:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484175232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53626us (54372us total) trans 4294972107 rc 0/0 00000100:00100000:3.0:1789760719.576335:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4818 00000100:00100000:3.0:1789760719.576349:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708789523530 time 708853597607 state 0x2 ref 0 00000100:00100000:3.0:1789760719.578812:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760719.578948:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484175360 00000100:00100000:0.0:1789760719.578961:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484175360 00000100:00100000:0.0:1789760719.579037:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708789523530 time 708853597607 state 0x0 ref 1 00000100:00100000:0.0:1789760719.579055:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708853858023 time 708853597607 state 0x1 ref 1 00000100:00100000:0.0:1789760719.579249:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708853858023 time 708910178190 state 0x0 ref 1 00000100:00100000:0.0:1789760719.579267:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484175360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760719.579336:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19619840, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:0.0:1789760719.579348:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.579368:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:0.0:1789760719.579373:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760719.579381:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19619840->19623935] (req 19619840->19623935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.579397:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19619840->19623935] (req 19619840->19623935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760719.613170:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972108 is committed 00002000:00010000:2.0:1789760719.613556:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19619840 -> 19623936 00010000:00010000:2.0:1789760719.613586:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19619840->19623935] (req 19619840->19623935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.613601:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19619840->19623935] (req 19619840->19623935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.613610:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19619840->19623935] (req 19619840->19623935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.613618:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760719.613623:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19619840->19623935] (req 19619840->19623935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.613634:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19619840->19623935] (req 19619840->19623935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.613659:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19619840->19623935] (req 19619840->19623935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.613666:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207cb7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19619840->19623935] (req 19619840->19623935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760719.613933:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484175360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34662us (35128us total) trans 4294972108 rc 0/0 00000100:00100000:2.0:1789760719.613954:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4819 00000100:00100000:2.0:1789760719.613968:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708853858023 time 708910178190 state 0x2 ref 0 00000100:00100000:1.0:1789760719.627590:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760719.630123:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484175488 00000100:00100000:3.0:1789760719.630142:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484175488 00000100:00100000:3.0:1789760719.630227:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708853858023 time 708910178190 state 0x0 ref 1 00000100:00100000:3.0:1789760719.630246:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708910438606 time 708910178190 state 0x1 ref 1 00000100:00100000:0.0:1789760719.630345:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708910438606 time 708961272451 state 0x0 ref 1 00000100:00100000:0.0:1789760719.630372:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484175488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760719.630451:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19623936, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:0.0:1789760719.630461:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.630495:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=708s v=(0 0) 00010000:00010000:0.0:1789760719.630500:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760719.630508:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19623936->19628031] (req 19623936->19628031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.630525:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19623936->19628031] (req 19623936->19628031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760719.685595:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19623936 -> 19628032 00010000:00010000:0.0:1789760719.685621:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19623936->19628031] (req 19623936->19628031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.685635:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19623936->19628031] (req 19623936->19628031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.685645:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19623936->19628031] (req 19623936->19628031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.685652:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760719.685658:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19623936->19628031] (req 19623936->19628031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.685669:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19623936->19628031] (req 19623936->19628031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.685694:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19623936->19628031] (req 19623936->19628031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.685701:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cbe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19623936->19628031] (req 19623936->19628031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.685824:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484175488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55451us (58238us total) trans 4294972109 rc 0/0 00000100:00100000:0.0:1789760719.685837:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4820 00000100:00100000:0.0:1789760719.685849:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708910438606 time 708961272451 state 0x2 ref 0 00000001:00080000:0.0:1789760719.685899:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972109 is committed 00000100:00100000:0.0:1789760719.697208:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760719.697330:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484175616 00000100:00100000:1.0:1789760719.697345:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484175616 00000100:00100000:1.0:1789760719.697440:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708910438606 time 708961272451 state 0x0 ref 1 00000100:00100000:1.0:1789760719.697463:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708961532867 time 708961272451 state 0x1 ref 1 00000100:00100000:1.0:1789760719.697509:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708961532867 time 709028439441 state 0x0 ref 1 00000100:00100000:1.0:1789760719.697525:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484175616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760719.697592:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19628032, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:1.0:1789760719.697602:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.697624:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:1.0:1789760719.697628:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760719.697634:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19628032->19632127] (req 19628032->19632127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.697648:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19628032->19632127] (req 19628032->19632127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760719.733401:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972110 is committed 00002000:00010000:0.0:1789760719.733476:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19628032 -> 19632128 00010000:00010000:0.0:1789760719.733509:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19628032->19632127] (req 19628032->19632127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.733525:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19628032->19632127] (req 19628032->19632127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.733540:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19628032->19632127] (req 19628032->19632127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.733555:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760719.733561:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19628032->19632127] (req 19628032->19632127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.733572:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19628032->19632127] (req 19628032->19632127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.733596:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19628032->19632127] (req 19628032->19632127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.733604:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396207cc5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19628032->19632127] (req 19628032->19632127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.733715:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484175616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36185us (36516us total) trans 4294972110 rc 0/0 00000100:00100000:0.0:1789760719.733728:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4821 00000100:00100000:0.0:1789760719.733740:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708961532867 time 709028439441 state 0x2 ref 0 00000100:00100000:3.0:1789760719.736552:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760719.737932:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484175744 00000100:00100000:1.0:1789760719.737948:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484175744 00000100:00100000:1.0:1789760719.738020:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 708961532867 time 709028439441 state 0x0 ref 1 00000100:00100000:1.0:1789760719.738042:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709028699857 time 709028439441 state 0x1 ref 1 00000100:00100000:1.0:1789760719.738165:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709028699857 time 709069095148 state 0x0 ref 1 00000100:00100000:1.0:1789760719.738183:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484175744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760719.738248:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19632128, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:1.0:1789760719.738271:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.738292:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:1.0:1789760719.738298:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760719.738454:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19632128->19636223] (req 19632128->19636223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760719.738473:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19632128->19636223] (req 19632128->19636223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760719.756955:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760719.778249:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972111 is committed 00002000:00010000:0.0:1789760719.778508:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19632128 -> 19636224 00010000:00010000:0.0:1789760719.778530:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19632128->19636223] (req 19632128->19636223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.778545:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19632128->19636223] (req 19632128->19636223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.778562:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19632128->19636223] (req 19632128->19636223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.778583:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760719.778589:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19632128->19636223] (req 19632128->19636223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.778600:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19632128->19636223] (req 19632128->19636223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.778628:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19632128->19636223] (req 19632128->19636223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.778637:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396207ccc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19632128->19636223] (req 19632128->19636223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.778773:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484175744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40587us (42224us total) trans 4294972111 rc 0/0 00000100:00100000:0.0:1789760719.778784:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4822 00000100:00100000:0.0:1789760719.778797:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709028699857 time 709069095148 state 0x2 ref 0 00000100:00100000:1.0:1789760719.791726:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760719.791932:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484175872 00000100:00100000:2.0:1789760719.791947:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484175872 00000100:00100000:2.0:1789760719.792022:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709028699857 time 709069095148 state 0x0 ref 1 00000100:00100000:2.0:1789760719.792134:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709069355564 time 709069095148 state 0x1 ref 1 00000100:00100000:2.0:1789760719.792170:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709069355564 time 709123101814 state 0x0 ref 1 00000100:00100000:2.0:1789760719.792185:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484175872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760719.792249:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19636224, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:2.0:1789760719.792427:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.792585:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:2.0:1789760719.792591:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760719.792599:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19636224->19640319] (req 19636224->19640319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.792617:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19636224->19640319] (req 19636224->19640319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760719.841963:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972112 is committed 00002000:00010000:0.0:1789760719.842158:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19636224 -> 19640320 00010000:00010000:0.0:1789760719.842178:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19636224->19640319] (req 19636224->19640319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.842201:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19636224->19640319] (req 19636224->19640319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.842210:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19636224->19640319] (req 19636224->19640319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.842218:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760719.842224:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19636224->19640319] (req 19636224->19640319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.842234:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19636224->19640319] (req 19636224->19640319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.842319:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19636224->19640319] (req 19636224->19640319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.842329:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396207cd3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19636224->19640319] (req 19636224->19640319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760719.842508:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484175872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50319us (50791us total) trans 4294972112 rc 0/0 00000100:00100000:0.0:1789760719.842520:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4823 00000100:00100000:0.0:1789760719.842533:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709069355564 time 709123101814 state 0x2 ref 0 00000100:00100000:1.0:1789760719.853902:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760719.854138:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484176000 00000100:00100000:0.0:1789760719.854222:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484176000 00000100:00100000:0.0:1789760719.854390:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709069355564 time 709123101814 state 0x0 ref 1 00000100:00100000:0.0:1789760719.854410:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709123362230 time 709123101814 state 0x1 ref 1 00000100:00100000:0.0:1789760719.854526:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709123362230 time 709185455174 state 0x0 ref 1 00000100:00100000:0.0:1789760719.854544:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484176000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760719.855108:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19640320, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:0.0:1789760719.855121:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.855140:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:0.0:1789760719.855146:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760719.855642:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19640320->19644415] (req 19640320->19644415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.855662:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19640320->19644415] (req 19640320->19644415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760719.925997:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972113 is committed 00002000:00010000:2.0:1789760719.926259:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19640320 -> 19644416 00010000:00010000:2.0:1789760719.926280:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19640320->19644415] (req 19640320->19644415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.926298:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19640320->19644415] (req 19640320->19644415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.926308:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19640320->19644415] (req 19640320->19644415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.926317:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760719.926322:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19640320->19644415] (req 19640320->19644415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.926333:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19640320->19644415] (req 19640320->19644415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.926360:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19640320->19644415] (req 19640320->19644415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760719.926369:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207cda lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19640320->19644415] (req 19640320->19644415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760719.926563:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484176000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 72018us (72671us total) trans 4294972113 rc 0/0 00000100:00100000:2.0:1789760719.926576:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4824 00000100:00100000:2.0:1789760719.926589:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709123362230 time 709185455174 state 0x2 ref 0 00000100:00100000:3.0:1789760719.935516:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760719.935772:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484176128 00000100:00100000:0.0:1789760719.935786:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484176128 00000100:00100000:0.0:1789760719.935874:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709123362230 time 709185455174 state 0x0 ref 1 00000100:00100000:0.0:1789760719.935895:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709185715590 time 709185455174 state 0x1 ref 1 00000100:00100000:0.0:1789760719.935938:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709185715590 time 709266867991 state 0x0 ref 1 00000100:00100000:0.0:1789760719.935956:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760719.936023:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19644416, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:0.0:1789760719.936035:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.936056:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:0.0:1789760719.936061:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760719.936126:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19644416->19648511] (req 19644416->19648511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760719.936146:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19644416->19648511] (req 19644416->19648511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760720.067744:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972114 is committed 00002000:00010000:3.0:1789760720.075515:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19644416 -> 19648512 00010000:00010000:3.0:1789760720.075543:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19644416->19648511] (req 19644416->19648511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760720.075559:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19644416->19648511] (req 19644416->19648511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760720.075631:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19644416->19648511] (req 19644416->19648511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760720.075641:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760720.075646:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19644416->19648511] (req 19644416->19648511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760720.075657:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19644416->19648511] (req 19644416->19648511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760720.075685:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19644416->19648511] (req 19644416->19648511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760720.075695:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207ce1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19644416->19648511] (req 19644416->19648511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760720.076392:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 140431us (140882us total) trans 4294972114 rc 0/0 00000100:00100000:3.0:1789760720.076407:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4825 00000100:00100000:3.0:1789760720.076420:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709185715590 time 709266867991 state 0x2 ref 0 00000100:00100000:1.0:1789760720.088732:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760720.088984:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484176256 00000100:00100000:0.0:1789760720.088998:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484176256 00000100:00100000:0.0:1789760720.089355:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709185715590 time 709266867991 state 0x0 ref 1 00000100:00100000:0.0:1789760720.089376:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709267128407 time 709266867991 state 0x1 ref 1 00000100:00100000:0.0:1789760720.089409:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709267128407 time 709420339479 state 0x0 ref 1 00000100:00100000:0.0:1789760720.089502:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760720.089628:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19648512, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:0.0:1789760720.089639:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.089666:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:0.0:1789760720.089671:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760720.089678:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19648512->19652607] (req 19648512->19652607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.089696:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19648512->19652607] (req 19648512->19652607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760720.090968:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:0.0:1789760720.157959:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972115 is committed 00002000:00010000:2.0:1789760720.158278:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19648512 -> 19652608 00010000:00010000:2.0:1789760720.158298:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19648512->19652607] (req 19648512->19652607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.158314:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19648512->19652607] (req 19648512->19652607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.158325:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19648512->19652607] (req 19648512->19652607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.158334:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760720.158340:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19648512->19652607] (req 19648512->19652607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.158351:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19648512->19652607] (req 19648512->19652607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.158383:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19648512->19652607] (req 19648512->19652607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.158392:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207ce8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19648512->19652607] (req 19648512->19652607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760720.158716:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69288us (69988us total) trans 4294972115 rc 0/0 00000100:00100000:2.0:1789760720.158728:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4826 00000100:00100000:2.0:1789760720.158740:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709267128407 time 709420339479 state 0x2 ref 0 00000100:00100000:1.0:1789760720.176320:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760720.176540:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484176384 00000100:00100000:2.0:1789760720.176554:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484176384 00000100:00100000:2.0:1789760720.176636:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709267128407 time 709420339479 state 0x0 ref 1 00000100:00100000:2.0:1789760720.176655:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709420599895 time 709420339479 state 0x1 ref 1 00000100:00100000:2.0:1789760720.176689:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709420599895 time 709507619933 state 0x0 ref 1 00000100:00100000:2.0:1789760720.176702:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760720.176777:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19652608, mtime: 0x6aad94cf, atime: 0x0, ctime: 0x6aad94cf, blocks: 0x9807 00010000:00010000:2.0:1789760720.176788:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.176807:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:2.0:1789760720.176814:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760720.176821:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19652608->19656703] (req 19652608->19656703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.176838:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19652608->19656703] (req 19652608->19656703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760720.289670:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972116 is committed 00002000:00010000:0.0:1789760720.290147:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19652608 -> 19656704 00002000:00010000:0.0:1789760720.290154:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760719 -> 1789760720 00002000:00010000:0.0:1789760720.290156:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760719 -> 1789760720 00010000:00010000:0.0:1789760720.290174:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19652608->19656703] (req 19652608->19656703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.290190:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19652608->19656703] (req 19652608->19656703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.290200:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19652608->19656703] (req 19652608->19656703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.290209:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760720.290214:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19652608->19656703] (req 19652608->19656703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.290226:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19652608->19656703] (req 19652608->19656703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.290257:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19652608->19656703] (req 19652608->19656703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.290266:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207cef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19652608->19656703] (req 19652608->19656703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760720.290377:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 113670us (114066us total) trans 4294972116 rc 0/0 00000100:00100000:0.0:1789760720.290388:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4827 00000100:00100000:0.0:1789760720.290399:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709420599895 time 709507619933 state 0x2 ref 0 00000100:00100000:0.0:1789760720.293448:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760720.293595:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484176512 00000100:00100000:2.0:1789760720.293608:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484176512 00000100:00100000:2.0:1789760720.293681:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709420599895 time 709507619933 state 0x0 ref 1 00000100:00100000:2.0:1789760720.293700:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709507880349 time 709507619933 state 0x1 ref 1 00000100:00100000:2.0:1789760720.293743:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709507880349 time 709624673420 state 0x0 ref 1 00000100:00100000:2.0:1789760720.293861:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760720.293928:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19656704, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:2.0:1789760720.293940:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.293960:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:2.0:1789760720.293965:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760720.293972:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19656704->19660799] (req 19656704->19660799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.293988:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19656704->19660799] (req 19656704->19660799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760720.332033:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972117 is committed 00002000:00010000:2.0:1789760720.332189:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19656704 -> 19660800 00010000:00010000:2.0:1789760720.332206:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19656704->19660799] (req 19656704->19660799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.332217:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19656704->19660799] (req 19656704->19660799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.332224:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19656704->19660799] (req 19656704->19660799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.332229:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760720.332234:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19656704->19660799] (req 19656704->19660799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.332241:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19656704->19660799] (req 19656704->19660799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.332526:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19656704->19660799] (req 19656704->19660799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.332535:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207cf6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19656704->19660799] (req 19656704->19660799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760720.332707:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38945us (39266us total) trans 4294972117 rc 0/0 00000100:00100000:2.0:1789760720.332722:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4828 00000100:00100000:2.0:1789760720.332734:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709507880349 time 709624673420 state 0x2 ref 0 00000100:00100000:1.0:1789760720.334993:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760720.335133:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484176640 00000100:00100000:3.0:1789760720.335146:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484176640 00000100:00100000:3.0:1789760720.335217:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709507880349 time 709624673420 state 0x0 ref 1 00000100:00100000:3.0:1789760720.335236:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709624933836 time 709624673420 state 0x1 ref 1 00000100:00100000:3.0:1789760720.335330:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709624933836 time 709666258522 state 0x0 ref 1 00000100:00100000:3.0:1789760720.335414:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760720.335470:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19660800, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:3.0:1789760720.335540:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760720.335559:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:3.0:1789760720.335564:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760720.335572:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19660800->19664895] (req 19660800->19664895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760720.335589:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19660800->19664895] (req 19660800->19664895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760720.366815:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972118 is committed 00002000:00010000:0.0:1789760720.367015:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19660800 -> 19664896 00010000:00010000:0.0:1789760720.367037:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19660800->19664895] (req 19660800->19664895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.367054:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19660800->19664895] (req 19660800->19664895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.367096:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19660800->19664895] (req 19660800->19664895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.367106:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760720.367112:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19660800->19664895] (req 19660800->19664895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.367130:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19660800->19664895] (req 19660800->19664895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.367157:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19660800->19664895] (req 19660800->19664895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.367167:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207cfd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19660800->19664895] (req 19660800->19664895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760720.367313:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31963us (32325us total) trans 4294972118 rc 0/0 00000100:00100000:0.0:1789760720.367326:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4829 00000100:00100000:0.0:1789760720.367339:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709624933836 time 709666258522 state 0x2 ref 0 00000100:00100000:1.0:1789760720.373180:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760720.374259:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484176768 00000100:00100000:0.0:1789760720.374275:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484176768 00000100:00100000:0.0:1789760720.374365:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709624933836 time 709666258522 state 0x0 ref 1 00000100:00100000:0.0:1789760720.374385:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709666518938 time 709666258522 state 0x1 ref 1 00000100:00100000:0.0:1789760720.374495:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709666518938 time 709705424218 state 0x0 ref 1 00000100:00100000:0.0:1789760720.374512:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760720.374578:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19664896, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:0.0:1789760720.374589:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.374609:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:0.0:1789760720.374614:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760720.374621:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19664896->19668991] (req 19664896->19668991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.374637:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19664896->19668991] (req 19664896->19668991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760720.429956:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972119 is committed 00002000:00010000:0.0:1789760720.430174:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19664896 -> 19668992 00010000:00010000:0.0:1789760720.430196:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19664896->19668991] (req 19664896->19668991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.430213:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19664896->19668991] (req 19664896->19668991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.430223:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19664896->19668991] (req 19664896->19668991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.430232:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760720.430238:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19664896->19668991] (req 19664896->19668991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.430249:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19664896->19668991] (req 19664896->19668991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.430582:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19664896->19668991] (req 19664896->19668991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.430592:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d04 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19664896->19668991] (req 19664896->19668991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760720.430734:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56218us (57560us total) trans 4294972119 rc 0/0 00000100:00100000:0.0:1789760720.430747:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4830 00000100:00100000:0.0:1789760720.430760:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709666518938 time 709705424218 state 0x2 ref 0 00000100:00100000:3.0:1789760720.435470:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760720.437211:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484176896 00000100:00100000:0.0:1789760720.437236:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484176896 00000100:00100000:0.0:1789760720.437339:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709666518938 time 709705424218 state 0x0 ref 1 00000100:00100000:0.0:1789760720.437360:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709705684634 time 709705424218 state 0x1 ref 1 00000100:00100000:0.0:1789760720.437390:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709705684634 time 709768320503 state 0x0 ref 1 00000100:00100000:0.0:1789760720.437408:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760720.437625:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19668992, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:0.0:1789760720.437637:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.437660:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:0.0:1789760720.437666:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760720.437674:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19668992->19673087] (req 19668992->19673087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.437698:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19668992->19673087] (req 19668992->19673087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760720.576348:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972120 is committed 00002000:00010000:2.0:1789760720.576691:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19668992 -> 19673088 00010000:00010000:2.0:1789760720.576713:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19668992->19673087] (req 19668992->19673087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.576728:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19668992->19673087] (req 19668992->19673087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.576737:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19668992->19673087] (req 19668992->19673087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.576745:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760720.576750:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19668992->19673087] (req 19668992->19673087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.576760:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19668992->19673087] (req 19668992->19673087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.576785:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19668992->19673087] (req 19668992->19673087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.576793:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207d0b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19668992->19673087] (req 19668992->19673087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760720.576936:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484176896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 139526us (141475us total) trans 4294972120 rc 0/0 00000100:00100000:2.0:1789760720.576949:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4831 00000100:00100000:2.0:1789760720.576962:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709705684634 time 709768320503 state 0x2 ref 0 00000100:00100000:3.0:1789760720.583710:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760720.583904:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484177024 00000100:00100000:2.0:1789760720.583918:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484177024 00000100:00100000:2.0:1789760720.583997:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709705684634 time 709768320503 state 0x0 ref 1 00000100:00100000:2.0:1789760720.584018:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709768580919 time 709768320503 state 0x1 ref 1 00000100:00100000:2.0:1789760720.584058:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709768580919 time 709914989100 state 0x0 ref 1 00000100:00100000:2.0:1789760720.584273:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484177024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760720.584342:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19673088, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:2.0:1789760720.584352:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.584372:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=709s v=(0 0) 00010000:00010000:2.0:1789760720.584378:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760720.584386:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19673088->19677183] (req 19673088->19677183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.584402:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19673088->19677183] (req 19673088->19677183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760720.663713:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972121 is committed 00002000:00010000:2.0:1789760720.666174:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19673088 -> 19677184 00010000:00010000:2.0:1789760720.666200:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19673088->19677183] (req 19673088->19677183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.666214:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19673088->19677183] (req 19673088->19677183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.666223:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19673088->19677183] (req 19673088->19677183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.666230:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760720.666236:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19673088->19677183] (req 19673088->19677183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.666245:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19673088->19677183] (req 19673088->19677183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.666272:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19673088->19677183] (req 19673088->19677183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.666280:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d12 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19673088->19677183] (req 19673088->19677183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760720.666511:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484177024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 82130us (82701us total) trans 4294972121 rc 0/0 00000100:00100000:2.0:1789760720.666527:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4832 00000100:00100000:2.0:1789760720.666539:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709768580919 time 709914989100 state 0x2 ref 0 00000100:00100000:3.0:1789760720.671377:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760720.671840:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484177152 00000100:00100000:2.0:1789760720.671853:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484177152 00000100:00100000:2.0:1789760720.671980:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709768580919 time 709914989100 state 0x0 ref 1 00000100:00100000:2.0:1789760720.671995:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709915249516 time 709914989100 state 0x1 ref 1 00000100:00100000:2.0:1789760720.672023:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709915249516 time 710002953898 state 0x0 ref 1 00000100:00100000:2.0:1789760720.672153:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484177152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760720.672219:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19677184, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:2.0:1789760720.672229:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.672249:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:2.0:1789760720.672386:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760720.672395:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19677184->19681279] (req 19677184->19681279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.672417:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19677184->19681279] (req 19677184->19681279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.781236:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760720.786259:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972122 is committed 00002000:00010000:2.0:1789760720.786684:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19677184 -> 19681280 00010000:00010000:2.0:1789760720.786705:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19677184->19681279] (req 19677184->19681279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.786723:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19677184->19681279] (req 19677184->19681279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.786732:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19677184->19681279] (req 19677184->19681279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.786740:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760720.786745:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19677184->19681279] (req 19677184->19681279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.786755:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19677184->19681279] (req 19677184->19681279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.786780:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19677184->19681279] (req 19677184->19681279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.786788:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d19 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19677184->19681279] (req 19677184->19681279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760720.786911:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484177152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 114756us (115542us total) trans 4294972122 rc 0/0 00000100:00100000:2.0:1789760720.786924:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4833 00000100:00100000:2.0:1789760720.786936:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709915249516 time 710002953898 state 0x2 ref 0 00000100:00100000:3.0:1789760720.791331:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760720.791422:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484177280 00000100:00100000:0.0:1789760720.791436:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484177280 00000100:00100000:0.0:1789760720.791533:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 709915249516 time 710002953898 state 0x0 ref 1 00000100:00100000:0.0:1789760720.791552:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710003214314 time 710002953898 state 0x1 ref 1 00000100:00100000:0.0:1789760720.791601:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710003214314 time 710122532120 state 0x0 ref 1 00000100:00100000:0.0:1789760720.791618:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484177280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760720.791687:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19681280, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:0.0:1789760720.791698:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.791718:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:0.0:1789760720.791723:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760720.791731:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19681280->19685375] (req 19681280->19685375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760720.791747:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19681280->19685375] (req 19681280->19685375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760720.861212:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972123 is committed 00002000:00010000:1.0:1789760720.861392:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19681280 -> 19685376 00010000:00010000:1.0:1789760720.861413:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19681280->19685375] (req 19681280->19685375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.861429:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19681280->19685375] (req 19681280->19685375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.861439:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19681280->19685375] (req 19681280->19685375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.861448:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760720.861458:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19681280->19685375] (req 19681280->19685375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.861483:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19681280->19685375] (req 19681280->19685375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.861510:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19681280->19685375] (req 19681280->19685375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.861517:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d20 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19681280->19685375] (req 19681280->19685375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760720.861755:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484177280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 70134us (70432us total) trans 4294972123 rc 0/0 00000100:00100000:1.0:1789760720.861767:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4834 00000100:00100000:1.0:1789760720.861779:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710003214314 time 710122532120 state 0x2 ref 0 00000100:00100000:3.0:1789760720.866493:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760720.866978:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484177408 00000100:00100000:1.0:1789760720.866993:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484177408 00000100:00100000:1.0:1789760720.867103:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710003214314 time 710122532120 state 0x0 ref 1 00000100:00100000:1.0:1789760720.867122:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710122792536 time 710122532120 state 0x1 ref 1 00000100:00100000:1.0:1789760720.867151:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710122792536 time 710198081791 state 0x0 ref 1 00000100:00100000:1.0:1789760720.867166:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484177408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760720.867233:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19685376, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:1.0:1789760720.867245:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.867281:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:1.0:1789760720.867286:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760720.867933:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19685376->19689471] (req 19685376->19689471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.867957:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19685376->19689471] (req 19685376->19689471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760720.933781:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972124 is committed 00002000:00010000:1.0:1789760720.933987:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19685376 -> 19689472 00010000:00010000:1.0:1789760720.934009:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19685376->19689471] (req 19685376->19689471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.934024:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19685376->19689471] (req 19685376->19689471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.934032:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19685376->19689471] (req 19685376->19689471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.934041:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760720.934047:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19685376->19689471] (req 19685376->19689471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.934057:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19685376->19689471] (req 19685376->19689471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.934109:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19685376->19689471] (req 19685376->19689471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760720.934119:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d27 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19685376->19689471] (req 19685376->19689471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760720.934361:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484177408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67190us (67882us total) trans 4294972124 rc 0/0 00000100:00100000:1.0:1789760720.934373:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4835 00000100:00100000:1.0:1789760720.934386:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710122792536 time 710198081791 state 0x2 ref 0 00000100:00100000:1.0:1789760720.937922:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760720.938209:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484177536 00000100:00100000:2.0:1789760720.938222:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484177536 00000100:00100000:2.0:1789760720.938361:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710122792536 time 710198081791 state 0x0 ref 1 00000100:00100000:2.0:1789760720.938379:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710198342207 time 710198081791 state 0x1 ref 1 00000100:00100000:2.0:1789760720.938439:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710198342207 time 710269368146 state 0x0 ref 1 00000100:00100000:2.0:1789760720.938454:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484177536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760720.938534:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19689472, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:2.0:1789760720.938545:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.938569:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:2.0:1789760720.938575:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760720.938582:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19689472->19693567] (req 19689472->19693567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760720.938599:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19689472->19693567] (req 19689472->19693567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760721.004855:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972125 is committed 00002000:00010000:1.0:1789760721.004969:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19689472 -> 19693568 00010000:00010000:1.0:1789760721.005140:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19689472->19693567] (req 19689472->19693567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.005157:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19689472->19693567] (req 19689472->19693567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.005167:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19689472->19693567] (req 19689472->19693567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.005175:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760721.005180:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19689472->19693567] (req 19689472->19693567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.005191:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19689472->19693567] (req 19689472->19693567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.005223:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19689472->19693567] (req 19689472->19693567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.005232:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207d2e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19689472->19693567] (req 19689472->19693567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760721.005583:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484177536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67125us (67669us total) trans 4294972125 rc 0/0 00000100:00100000:1.0:1789760721.005594:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4836 00000100:00100000:1.0:1789760721.005607:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710198342207 time 710269368146 state 0x2 ref 0 00000100:00100000:1.0:1789760721.009052:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760721.009191:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484177664 00000100:00100000:2.0:1789760721.009204:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484177664 00000100:00100000:2.0:1789760721.009298:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710198342207 time 710269368146 state 0x0 ref 1 00000100:00100000:2.0:1789760721.009318:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710269628562 time 710269368146 state 0x1 ref 1 00000100:00100000:2.0:1789760721.009353:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710269628562 time 710340283371 state 0x0 ref 1 00000100:00100000:2.0:1789760721.009370:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484177664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760721.009438:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19693568, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:2.0:1789760721.009449:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.009479:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:2.0:1789760721.009484:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760721.009491:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19693568->19697663] (req 19693568->19697663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.009509:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19693568->19697663] (req 19693568->19697663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760721.085574:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972126 is committed 00002000:00010000:1.0:1789760721.086767:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19693568 -> 19697664 00010000:00010000:1.0:1789760721.086793:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19693568->19697663] (req 19693568->19697663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.086809:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19693568->19697663] (req 19693568->19697663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.086820:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19693568->19697663] (req 19693568->19697663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.086829:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760721.086834:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19693568->19697663] (req 19693568->19697663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.086850:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19693568->19697663] (req 19693568->19697663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.086888:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19693568->19697663] (req 19693568->19697663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.086896:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396207d35 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19693568->19697663] (req 19693568->19697663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760721.087031:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484177664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77658us (77988us total) trans 4294972126 rc 0/0 00000100:00100000:1.0:1789760721.087044:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4837 00000100:00100000:1.0:1789760721.087057:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710269628562 time 710340283371 state 0x2 ref 0 00000100:00100000:3.0:1789760721.093233:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760721.093924:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484177792 00000100:00100000:1.0:1789760721.093940:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484177792 00000100:00100000:1.0:1789760721.094020:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710269628562 time 710340283371 state 0x0 ref 1 00000100:00100000:1.0:1789760721.094045:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710340543787 time 710340283371 state 0x1 ref 1 00000100:00100000:1.0:1789760721.094731:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710340543787 time 710425657041 state 0x0 ref 1 00000100:00100000:1.0:1789760721.094754:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484177792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760721.094826:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19697664, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:1.0:1789760721.094837:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.094857:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:1.0:1789760721.094862:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760721.094870:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19697664->19701759] (req 19697664->19701759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.094886:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19697664->19701759] (req 19697664->19701759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760721.187793:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972127 is committed 00002000:00010000:1.0:1789760721.194513:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19697664 -> 19701760 00010000:00010000:1.0:1789760721.194542:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19697664->19701759] (req 19697664->19701759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.194559:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19697664->19701759] (req 19697664->19701759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.194570:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19697664->19701759] (req 19697664->19701759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.194579:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760721.194585:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19697664->19701759] (req 19697664->19701759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.194596:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19697664->19701759] (req 19697664->19701759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.194624:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19697664->19701759] (req 19697664->19701759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.194634:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207d3c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19697664->19701759] (req 19697664->19701759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760721.194784:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484177792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 100030us (101562us total) trans 4294972127 rc 0/0 00000100:00100000:1.0:1789760721.194796:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4838 00000100:00100000:1.0:1789760721.194809:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710340543787 time 710425657041 state 0x2 ref 0 00000100:00100000:0.0:1789760721.203973:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760721.204194:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484177920 00000100:00100000:1.0:1789760721.204208:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484177920 00000100:00100000:1.0:1789760721.204291:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710340543787 time 710425657041 state 0x0 ref 1 00000100:00100000:1.0:1789760721.204309:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710425917457 time 710425657041 state 0x1 ref 1 00000100:00100000:1.0:1789760721.204340:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710425917457 time 710535271442 state 0x0 ref 1 00000100:00100000:1.0:1789760721.204355:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484177920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760721.204698:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19701760, mtime: 0x6aad94d0, atime: 0x0, ctime: 0x6aad94d0, blocks: 0x9807 00010000:00010000:3.0:1789760721.204710:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.204729:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:3.0:1789760721.204733:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760721.204741:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19701760->19705855] (req 19701760->19705855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.204757:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19701760->19705855] (req 19701760->19705855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760721.272338:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19701760 -> 19705856 00002000:00010000:0.0:1789760721.272354:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760720 -> 1789760721 00002000:00010000:0.0:1789760721.272357:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760720 -> 1789760721 00010000:00010000:0.0:1789760721.272683:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19701760->19705855] (req 19701760->19705855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.272701:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19701760->19705855] (req 19701760->19705855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.272712:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19701760->19705855] (req 19701760->19705855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.272720:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760721.272726:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19701760->19705855] (req 19701760->19705855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.272739:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19701760->19705855] (req 19701760->19705855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.272768:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19701760->19705855] (req 19701760->19705855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.272778:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207d43 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19701760->19705855] (req 19701760->19705855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760721.272915:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484177920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68557us (68949us total) trans 4294972128 rc 0/0 00000100:00100000:0.0:1789760721.272932:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4839 00000100:00100000:0.0:1789760721.272945:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710425917457 time 710535271442 state 0x2 ref 0 00000001:00080000:0.0:1789760721.273746:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972128 is committed 00000100:00100000:3.0:1789760721.279611:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760721.280252:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484178048 00000100:00100000:2.0:1789760721.280267:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484178048 00000100:00100000:2.0:1789760721.280354:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710425917457 time 710535271442 state 0x0 ref 1 00000100:00100000:2.0:1789760721.280503:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710535531858 time 710535271442 state 0x1 ref 1 00000100:00100000:2.0:1789760721.280544:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710535531858 time 710611473821 state 0x0 ref 1 00000100:00100000:2.0:1789760721.280560:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484178048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760721.280626:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19705856, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:2.0:1789760721.280637:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.280663:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:2.0:1789760721.280668:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760721.280676:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19705856->19709951] (req 19705856->19709951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.280693:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19705856->19709951] (req 19705856->19709951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760721.292985:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c10e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508911104:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760721.293106:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760721.293230:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508911104 00000100:00100000:0.0:1789760721.293240:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508911104 00000100:00100000:0.0:1789760721.296003:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 364 00000100:00100000:0.0:1789760721.296034:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508911104:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:0.0:1789760721.296099:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:0.0:1789760721.296216:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508911104:12345-0@lo:13:osp-pre-1-1.0 Request processed in 198us (3108us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760721.296224:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 364 00000100:00080000:0.0:1789760721.299752:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:0.0:1789760721.300505:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c10700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508911232:0@lo:400:kworker.0 00000100:00100000:0.0:1789760721.300598:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760721.300651:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c12a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508911744:0@lo:400:kworker.0 00000100:00100000:0.0:1789760721.300681:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760721.300702:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e85180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508912256:0@lo:400:kworker.0 00000100:00100000:0.0:1789760721.300727:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760721.300755:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [2->3] 00000100:00100000:0.0:1789760721.300761:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c11180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508911616:0@lo:400:kworker.0 00000100:00100000:0.0:1789760721.300789:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760721.300807:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508912128:0@lo:400:kworker.0 00000100:00100000:0.0:1789760721.300831:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760721.300847:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508912640:0@lo:400:kworker.0 00000100:00100000:0.0:1789760721.300870:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760721.300903:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508911232 00000100:00100000:0.0:1789760721.300913:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508911232 00000100:00100000:0.0:1789760721.300978:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 470 00000100:00100000:0.0:1789760721.300989:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b4850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508911232:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760721.301050:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b4850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508911232:12345-0@lo:400:kworker.0 Request processed in 63us (451us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760721.301058:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 470 00000100:00100000:0.0:1789760721.301128:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508912256 00000100:00100000:0.0:1789760721.301132:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508912256 00000100:00100000:0.0:1789760721.301167:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508912128 00000100:00100000:0.0:1789760721.301170:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508912128 00000100:00100000:0.0:1789760721.301191:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508912640 00000100:00100000:0.0:1789760721.301194:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508912640 00000100:00100000:0.0:1789760721.301213:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 309 00000100:00100000:0.0:1789760721.301219:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508912256:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.301755:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c11500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508911488:0@lo:400:kworker.0 00000100:00100000:0.0:1789760721.302523:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508912256:12345-0@lo:400:kworker.0 Request processed in 1298us (1789us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760721.302535:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 309 00000100:00100000:0.0:1789760721.302566:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 310 00000100:00100000:0.0:1789760721.302582:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508912128:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760721.303216:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c11f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508911360:0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.303632:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760721.303668:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c13b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0000_UUID:3479:1876699508912000:0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.303697:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760721.303715:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0001_UUID:3479:1876699508912512:0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.303738:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760721.303780:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508911744 00000100:00100000:3.0:1789760721.303789:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508911744 00000100:00100000:3.0:1789760721.303829:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508911488 00000100:00100000:3.0:1789760721.303832:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508911488 00000100:00100000:3.0:1789760721.303849:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 423 00000100:00100000:3.0:1789760721.303857:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508911744:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.303912:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508911744:12345-0@lo:400:kworker.0 Request processed in 55us (3229us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760721.303919:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 423 00000100:00100000:3.0:1789760721.303937:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 424 00000100:00100000:3.0:1789760721.303943:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508911488:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.303974:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508911488:12345-0@lo:400:kworker.0 Request processed in 31us (344us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760721.303980:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 424 00000100:00100000:3.0:1789760721.303997:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508911616 00000100:00100000:3.0:1789760721.304000:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508911616 00000100:00100000:3.0:1789760721.304022:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508912000 00000100:00100000:3.0:1789760721.304024:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508912000 00000100:00100000:3.0:1789760721.304043:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508912512 00000100:00100000:3.0:1789760721.304046:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508912512 00000100:00100000:3.0:1789760721.304085:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 875 00000100:00100000:3.0:1789760721.304091:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508911616:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.304120:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508911616:12345-0@lo:400:kworker.0 Request processed in 30us (3331us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760721.304126:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 875 00000100:00100000:3.0:1789760721.304136:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 876 00000100:00100000:3.0:1789760721.304141:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0000_UUID+5:3479:x1876699508912000:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.304166:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0000_UUID+5:3479:x1876699508912000:12345-0@lo:400:kworker.0 Request processed in 24us (467us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760721.304171:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 876 00000100:00100000:3.0:1789760721.304182:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 877 00000100:00100000:3.0:1789760721.304187:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3479:x1876699508912512:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.304210:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3479:x1876699508912512:12345-0@lo:400:kworker.0 Request processed in 23us (471us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760721.304216:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 877 00000100:00100000:2.0:1789760721.308389:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c10e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508911104:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760721.308434:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:2.0:1789760721.308450:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508912768:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760721.308554:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760721.308638:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508912768 00000100:00100000:2.0:1789760721.308647:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508912768 00000100:00100000:2.0:1789760721.308702:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 365 00000100:00100000:2.0:1789760721.308711:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+6:3480:x1876699508912768:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1789760721.308737:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760721.308782:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+6:3480:x1876699508912768:12345-0@lo:13:osp-pre-1-0.0 Request processed in 72us (230us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760721.308788:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 365 00000100:00100000:2.0:1789760721.308825:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508912768:0@lo:13:osp-pre-1-0.0 00000100:00100000:3.0:1789760721.309114:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c11500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508911488:0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.309140:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c13b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0000_UUID:3479:1876699508912000:0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.309154:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0001_UUID:3479:1876699508912512:0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.313012:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 311 00000100:00100000:3.0:1789760721.313044:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508912640:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760721.313191:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508912640:12345-0@lo:400:kworker.0 Request processed in 162us (12319us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760721.313199:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 311 00000100:00080000:3.0:1789760721.313266:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:3.0:1789760721.313272:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:3.0:1789760721.313276:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760721.314928:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c11180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508911616:0@lo:400:kworker.0 00000100:00100000:2.0:1789760721.314952:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508912640:0@lo:400:kworker.0 00000100:00100000:2.0:1789760721.314965:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c10700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508911232:0@lo:400:kworker.0 00000100:00100000:2.0:1789760721.314978:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c12a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508911744:0@lo:400:kworker.0 00000100:00100000:2.0:1789760721.314990:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e85180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508912256:0@lo:400:kworker.0 00000100:00100000:1.0:1789760721.322715:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760721.322793:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c12680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508911872:0@lo:400:kworker.0 00000100:00100000:1.0:1789760721.322832:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760721.322852:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e87b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508912384:0@lo:400:kworker.0 00000100:00100000:1.0:1789760721.322875:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760721.322951:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508911360 00000100:00100000:1.0:1789760721.322962:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508911360 00000100:00100000:1.0:1789760721.323020:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508911872 00000100:00100000:1.0:1789760721.323023:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508911872 00000100:00100000:1.0:1789760721.323051:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508912384 00000100:00100000:1.0:1789760721.323054:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508912384 00000100:00100000:1.0:1789760721.323091:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 878 00000100:00100000:1.0:1789760721.323100:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508911360:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760721.323165:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508911360:12345-0@lo:400:kworker.0 Request processed in 66us (462us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760721.323172:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 878 00000100:00100000:1.0:1789760721.323196:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 879 00000100:00100000:1.0:1789760721.323202:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508911872:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760721.323228:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508911872:12345-0@lo:400:kworker.0 Request processed in 27us (395us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760721.323233:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 879 00000100:00100000:1.0:1789760721.323246:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 880 00000100:00100000:1.0:1789760721.323251:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508912384:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760721.323275:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508912384:12345-0@lo:400:kworker.0 Request processed in 24us (398us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760721.323280:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 880 00000100:00100000:1.0:1789760721.324707:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c11f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508911360:0@lo:400:kworker.0 00000100:00100000:1.0:1789760721.324727:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c12680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508911872:0@lo:400:kworker.0 00000100:00100000:1.0:1789760721.324740:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e87b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508912384:0@lo:400:kworker.0 00000100:00080000:1.0:1789760721.324782:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:1.0:1789760721.329853:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508912128:12345-0@lo:400:kworker.0 Request processed in 27266us (29008us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760721.329874:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 310 00000100:00100000:1.0:1789760721.329961:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508912128:0@lo:400:kworker.0 00000001:00080000:3.0:1789760721.398248:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972129 is committed 00002000:00010000:2.0:1789760721.398521:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19705856 -> 19709952 00010000:00010000:2.0:1789760721.398542:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19705856->19709951] (req 19705856->19709951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.398558:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19705856->19709951] (req 19705856->19709951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.398567:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19705856->19709951] (req 19705856->19709951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.398575:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760721.398580:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19705856->19709951] (req 19705856->19709951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.398591:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19705856->19709951] (req 19705856->19709951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.398616:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19705856->19709951] (req 19705856->19709951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.398625:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d4a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19705856->19709951] (req 19705856->19709951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760721.398755:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484178048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 118193us (119153us total) trans 4294972129 rc 0/0 00000100:00100000:2.0:1789760721.398766:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4840 00000100:00100000:2.0:1789760721.398778:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710535531858 time 710611473821 state 0x2 ref 0 00000100:00100000:1.0:1789760721.402445:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760721.402552:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484178176 00000100:00100000:2.0:1789760721.402567:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484178176 00000100:00100000:2.0:1789760721.402642:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710535531858 time 710611473821 state 0x0 ref 1 00000100:00100000:2.0:1789760721.402663:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710611734237 time 710611473821 state 0x1 ref 1 00000100:00100000:2.0:1789760721.402753:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710611734237 time 710733682886 state 0x0 ref 1 00000100:00100000:2.0:1789760721.402768:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484178176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760721.402832:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19709952, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:2.0:1789760721.402843:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.402863:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:2.0:1789760721.402869:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760721.402877:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19709952->19714047] (req 19709952->19714047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.402894:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19709952->19714047] (req 19709952->19714047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760721.490226:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972130 is committed 00002000:00010000:2.0:1789760721.490492:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19709952 -> 19714048 00010000:00010000:2.0:1789760721.490513:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19709952->19714047] (req 19709952->19714047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.490529:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19709952->19714047] (req 19709952->19714047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.490539:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19709952->19714047] (req 19709952->19714047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.490546:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760721.490815:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19709952->19714047] (req 19709952->19714047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.490829:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19709952->19714047] (req 19709952->19714047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.490857:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19709952->19714047] (req 19709952->19714047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.490866:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d51 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19709952->19714047] (req 19709952->19714047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760721.491001:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484178176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 88228us (88562us total) trans 4294972130 rc 0/0 00000100:00100000:2.0:1789760721.491017:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4841 00000100:00100000:2.0:1789760721.491030:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710611734237 time 710733682886 state 0x2 ref 0 00000100:00100000:1.0:1789760721.495739:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760721.495859:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484178304 00000100:00100000:0.0:1789760721.495873:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484178304 00000100:00100000:0.0:1789760721.495958:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710611734237 time 710733682886 state 0x0 ref 1 00000100:00100000:0.0:1789760721.495980:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710733943302 time 710733682886 state 0x1 ref 1 00000100:00100000:0.0:1789760721.496046:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710733943302 time 710826976053 state 0x0 ref 1 00000100:00100000:0.0:1789760721.496062:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484178304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760721.496148:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19714048, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:0.0:1789760721.496158:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.496179:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:0.0:1789760721.496185:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760721.496193:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19714048->19718143] (req 19714048->19718143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.496209:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19714048->19718143] (req 19714048->19718143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760721.532907:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19714048 -> 19718144 00010000:00010000:0.0:1789760721.532935:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19714048->19718143] (req 19714048->19718143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.532949:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19714048->19718143] (req 19714048->19718143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.532959:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19714048->19718143] (req 19714048->19718143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.532973:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760721.532978:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19714048->19718143] (req 19714048->19718143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.532988:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19714048->19718143] (req 19714048->19718143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.533013:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19714048->19718143] (req 19714048->19718143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.533022:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207d58 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19714048->19718143] (req 19714048->19718143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760721.533204:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484178304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37139us (37472us total) trans 4294972131 rc 0/0 00000100:00100000:0.0:1789760721.533218:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4842 00000100:00100000:0.0:1789760721.533232:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710733943302 time 710826976053 state 0x2 ref 0 00000001:00080000:1.0:1789760721.533693:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972131 is committed 00000100:00100000:1.0:1789760721.536827:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760721.536920:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484178432 00000100:00100000:0.0:1789760721.536934:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484178432 00000100:00100000:0.0:1789760721.537015:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710733943302 time 710826976053 state 0x0 ref 1 00000100:00100000:0.0:1789760721.537034:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710827236469 time 710826976053 state 0x1 ref 1 00000100:00100000:0.0:1789760721.537101:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710827236469 time 710868030610 state 0x0 ref 1 00000100:00100000:0.0:1789760721.537121:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484178432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760721.537186:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19718144, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:0.0:1789760721.537197:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.537218:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:0.0:1789760721.537223:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760721.537231:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19718144->19722239] (req 19718144->19722239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.537259:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19718144->19722239] (req 19718144->19722239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760721.590950:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972132 is committed 00002000:00010000:0.0:1789760721.591812:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19718144 -> 19722240 00010000:00010000:0.0:1789760721.591834:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19718144->19722239] (req 19718144->19722239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.591849:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19718144->19722239] (req 19718144->19722239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.591858:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19718144->19722239] (req 19718144->19722239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.591866:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760721.591872:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19718144->19722239] (req 19718144->19722239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.591883:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19718144->19722239] (req 19718144->19722239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.591907:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19718144->19722239] (req 19718144->19722239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.591916:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d5f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19718144->19722239] (req 19718144->19722239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760721.592763:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484178432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55433us (55736us total) trans 4294972132 rc 0/0 00000100:00100000:0.0:1789760721.592777:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4843 00000100:00100000:0.0:1789760721.592790:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710827236469 time 710868030610 state 0x2 ref 0 00000100:00100000:1.0:1789760721.599259:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760721.599369:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484178560 00000100:00100000:2.0:1789760721.599386:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484178560 00000100:00100000:2.0:1789760721.599478:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710827236469 time 710868030610 state 0x0 ref 1 00000100:00100000:2.0:1789760721.599497:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710868291026 time 710868030610 state 0x1 ref 1 00000100:00100000:2.0:1789760721.599536:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710868291026 time 710930466871 state 0x0 ref 1 00000100:00100000:2.0:1789760721.599559:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484178560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760721.599622:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19722240, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:2.0:1789760721.599632:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.599652:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=710s v=(0 0) 00010000:00010000:2.0:1789760721.599658:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760721.599665:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19722240->19726335] (req 19722240->19726335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.599681:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19722240->19726335] (req 19722240->19726335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760721.671713:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972133 is committed 00002000:00010000:3.0:1789760721.671946:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19722240 -> 19726336 00010000:00010000:3.0:1789760721.671967:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19722240->19726335] (req 19722240->19726335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.671984:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19722240->19726335] (req 19722240->19726335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.671994:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19722240->19726335] (req 19722240->19726335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.672002:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760721.672008:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19722240->19726335] (req 19722240->19726335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.672114:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19722240->19726335] (req 19722240->19726335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.672144:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19722240->19726335] (req 19722240->19726335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.672153:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207d66 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19722240->19726335] (req 19722240->19726335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760721.672474:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484178560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 72909us (73213us total) trans 4294972133 rc 0/0 00000100:00100000:3.0:1789760721.672487:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4844 00000100:00100000:3.0:1789760721.672500:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710868291026 time 710930466871 state 0x2 ref 0 00000100:00100000:1.0:1789760721.676808:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760721.676933:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484178688 00000100:00100000:2.0:1789760721.676947:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484178688 00000100:00100000:2.0:1789760721.677167:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710868291026 time 710930466871 state 0x0 ref 1 00000100:00100000:2.0:1789760721.677189:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710930727287 time 710930466871 state 0x1 ref 1 00000100:00100000:2.0:1789760721.677282:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710930727287 time 711008211349 state 0x0 ref 1 00000100:00100000:2.0:1789760721.677298:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484178688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760721.677540:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19726336, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:2.0:1789760721.677551:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.677571:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760721.677575:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760721.677583:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19726336->19730431] (req 19726336->19730431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.677600:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19726336->19730431] (req 19726336->19730431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760721.726136:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19726336 -> 19730432 00010000:00010000:3.0:1789760721.726165:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19726336->19730431] (req 19726336->19730431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.726181:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19726336->19730431] (req 19726336->19730431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.726191:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19726336->19730431] (req 19726336->19730431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.726199:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760721.726205:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19726336->19730431] (req 19726336->19730431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.726220:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19726336->19730431] (req 19726336->19730431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.726673:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19726336->19730431] (req 19726336->19730431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.726684:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396207d6d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19726336->19730431] (req 19726336->19730431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760721.726888:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484178688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49587us (50086us total) trans 4294972134 rc 0/0 00000100:00100000:3.0:1789760721.726901:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4845 00000100:00100000:3.0:1789760721.726915:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710930727287 time 711008211349 state 0x2 ref 0 00000100:00100000:1.0:1789760721.729899:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760721.730007:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484178816 00000100:00100000:2.0:1789760721.730021:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484178816 00000100:00100000:2.0:1789760721.730132:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 710930727287 time 711008211349 state 0x0 ref 1 00000100:00100000:2.0:1789760721.730150:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711008471765 time 711008211349 state 0x1 ref 1 00000100:00100000:2.0:1789760721.730172:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711008471765 time 711061103308 state 0x0 ref 1 00000100:00100000:2.0:1789760721.730185:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484178816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760721.730262:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19730432, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:2.0:1789760721.730272:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.730292:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760721.730297:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760721.730303:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19730432->19734527] (req 19730432->19734527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.730319:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19730432->19734527] (req 19730432->19734527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760721.730848:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972134 is committed 00000001:00080000:2.0:1789760721.777195:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972135 is committed 00002000:00010000:3.0:1789760721.777303:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19730432 -> 19734528 00010000:00010000:3.0:1789760721.777324:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19730432->19734527] (req 19730432->19734527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.777339:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19730432->19734527] (req 19730432->19734527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.777350:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19730432->19734527] (req 19730432->19734527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.777357:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760721.777364:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19730432->19734527] (req 19730432->19734527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.777375:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19730432->19734527] (req 19730432->19734527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.777401:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19730432->19734527] (req 19730432->19734527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.777408:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207d74 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19730432->19734527] (req 19730432->19734527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760721.777547:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484178816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47357us (47653us total) trans 4294972135 rc 0/0 00000100:00100000:3.0:1789760721.777560:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4846 00000100:00100000:3.0:1789760721.777574:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711008471765 time 711061103308 state 0x2 ref 0 00000100:00100000:1.0:1789760721.780778:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760721.780846:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484178944 00000100:00100000:0.0:1789760721.780860:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484178944 00000100:00100000:0.0:1789760721.780942:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711008471765 time 711061103308 state 0x0 ref 1 00000100:00100000:0.0:1789760721.780961:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711061363724 time 711061103308 state 0x1 ref 1 00000100:00100000:0.0:1789760721.781032:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711061363724 time 711111962771 state 0x0 ref 1 00000100:00100000:0.0:1789760721.781047:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484178944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760721.781158:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19734528, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:0.0:1789760721.781170:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.781190:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:0.0:1789760721.781195:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760721.781202:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19734528->19738623] (req 19734528->19738623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.781220:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19734528->19738623] (req 19734528->19738623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760721.798970:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972136 is committed 00002000:00010000:0.0:1789760721.799161:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19734528 -> 19738624 00010000:00010000:0.0:1789760721.799182:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19734528->19738623] (req 19734528->19738623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.799197:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19734528->19738623] (req 19734528->19738623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.799208:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19734528->19738623] (req 19734528->19738623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.799216:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760721.799221:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19734528->19738623] (req 19734528->19738623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.799233:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19734528->19738623] (req 19734528->19738623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.800169:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19734528->19738623] (req 19734528->19738623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.800185:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207d7b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19734528->19738623] (req 19734528->19738623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760721.800332:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484178944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 19281us (19561us total) trans 4294972136 rc 0/0 00000100:00100000:0.0:1789760721.800341:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4847 00000100:00100000:0.0:1789760721.800352:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711061363724 time 711111962771 state 0x2 ref 0 00010000:00010000:2.0:1789760721.805351:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760721.806288:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760721.806413:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484179072 00000100:00100000:3.0:1789760721.806429:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484179072 00000100:00100000:3.0:1789760721.806518:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711061363724 time 711111962771 state 0x0 ref 1 00000100:00100000:3.0:1789760721.806538:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711112223187 time 711111962771 state 0x1 ref 1 00000100:00100000:3.0:1789760721.806751:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711112223187 time 711137680253 state 0x0 ref 1 00000100:00100000:3.0:1789760721.806764:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484179072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760721.806823:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19738624, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:3.0:1789760721.806831:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.806848:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:3.0:1789760721.806907:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760721.806916:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19738624->19742719] (req 19738624->19742719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.806934:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19738624->19742719] (req 19738624->19742719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760721.852142:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972137 is committed 00002000:00010000:2.0:1789760721.852363:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19738624 -> 19742720 00010000:00010000:2.0:1789760721.852386:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19738624->19742719] (req 19738624->19742719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.852402:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19738624->19742719] (req 19738624->19742719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.852413:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19738624->19742719] (req 19738624->19742719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.852421:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760721.852426:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19738624->19742719] (req 19738624->19742719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.852437:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19738624->19742719] (req 19738624->19742719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.852473:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19738624->19742719] (req 19738624->19742719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.852482:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207d82 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19738624->19742719] (req 19738624->19742719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760721.852639:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484179072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45870us (46389us total) trans 4294972137 rc 0/0 00000100:00100000:2.0:1789760721.852650:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4848 00000100:00100000:2.0:1789760721.852663:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711112223187 time 711137680253 state 0x2 ref 0 00000100:00100000:2.0:1789760721.860671:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760721.860748:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484179200 00000100:00100000:3.0:1789760721.860762:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484179200 00000100:00100000:3.0:1789760721.860830:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711112223187 time 711137680253 state 0x0 ref 1 00000100:00100000:3.0:1789760721.860848:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711137940669 time 711137680253 state 0x1 ref 1 00000100:00100000:3.0:1789760721.860913:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711137940669 time 711191843112 state 0x0 ref 1 00000100:00100000:3.0:1789760721.860927:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484179200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760721.860986:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19742720, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:3.0:1789760721.860997:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.861018:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:3.0:1789760721.861023:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760721.861030:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19742720->19746815] (req 19742720->19746815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760721.861048:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19742720->19746815] (req 19742720->19746815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760721.933799:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972138 is committed 00002000:00010000:1.0:1789760721.934110:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19742720 -> 19746816 00010000:00010000:1.0:1789760721.934132:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19742720->19746815] (req 19742720->19746815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.934148:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19742720->19746815] (req 19742720->19746815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.934158:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19742720->19746815] (req 19742720->19746815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.934166:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760721.934172:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19742720->19746815] (req 19742720->19746815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.934188:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19742720->19746815] (req 19742720->19746815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.934215:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19742720->19746815] (req 19742720->19746815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760721.934224:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207d89 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19742720->19746815] (req 19742720->19746815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760721.934341:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484179200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73410us (73676us total) trans 4294972138 rc 0/0 00000100:00100000:1.0:1789760721.934352:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4849 00000100:00100000:1.0:1789760721.934365:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711137940669 time 711191843112 state 0x2 ref 0 00000100:00100000:1.0:1789760721.937399:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760721.937665:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484179328 00000100:00100000:2.0:1789760721.937680:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484179328 00000100:00100000:2.0:1789760721.937761:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711137940669 time 711191843112 state 0x0 ref 1 00000100:00100000:2.0:1789760721.937789:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711192103528 time 711191843112 state 0x1 ref 1 00000100:00100000:2.0:1789760721.937833:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711192103528 time 711268763138 state 0x0 ref 1 00000100:00100000:2.0:1789760721.937849:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484179328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760721.937914:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19746816, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:2.0:1789760721.937925:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.937944:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760721.937949:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760721.937957:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19746816->19750911] (req 19746816->19750911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.937973:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19746816->19750911] (req 19746816->19750911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760721.984727:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972139 is committed 00002000:00010000:0.0:1789760721.986773:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19746816 -> 19750912 00010000:00010000:0.0:1789760721.986798:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19746816->19750911] (req 19746816->19750911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.987283:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19746816->19750911] (req 19746816->19750911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.987293:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19746816->19750911] (req 19746816->19750911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.987300:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760721.987305:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19746816->19750911] (req 19746816->19750911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.987316:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19746816->19750911] (req 19746816->19750911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.987343:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19746816->19750911] (req 19746816->19750911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760721.987352:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207d90 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19746816->19750911] (req 19746816->19750911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760721.987497:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484179328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49645us (50104us total) trans 4294972139 rc 0/0 00000100:00100000:0.0:1789760721.987511:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4850 00000100:00100000:0.0:1789760721.987523:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711192103528 time 711268763138 state 0x2 ref 0 00000100:00100000:0.0:1789760721.992178:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760721.992424:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484179456 00000100:00100000:2.0:1789760721.992435:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484179456 00000100:00100000:2.0:1789760721.992518:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711192103528 time 711268763138 state 0x0 ref 1 00000100:00100000:2.0:1789760721.992539:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711269023554 time 711268763138 state 0x1 ref 1 00000100:00100000:2.0:1789760721.992663:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711269023554 time 711323591617 state 0x0 ref 1 00000100:00100000:2.0:1789760721.992678:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484179456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760721.992747:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19750912, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:2.0:1789760721.992758:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.992777:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760721.992783:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760721.992790:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19750912->19755007] (req 19750912->19755007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760721.992817:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19750912->19755007] (req 19750912->19755007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760722.028183:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972140 is committed 00002000:00010000:2.0:1789760722.028308:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19750912 -> 19755008 00010000:00010000:2.0:1789760722.028328:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19750912->19755007] (req 19750912->19755007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.028343:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19750912->19755007] (req 19750912->19755007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.028353:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19750912->19755007] (req 19750912->19755007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.028361:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760722.028367:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19750912->19755007] (req 19750912->19755007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.028377:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19750912->19755007] (req 19750912->19755007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.028400:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19750912->19755007] (req 19750912->19755007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.028408:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207d97 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19750912->19755007] (req 19750912->19755007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760722.028906:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484179456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36223us (36735us total) trans 4294972140 rc 0/0 00000100:00100000:2.0:1789760722.028919:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4851 00000100:00100000:2.0:1789760722.028932:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711269023554 time 711323591617 state 0x2 ref 0 00000100:00100000:2.0:1789760722.036930:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760722.037040:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484179584 00000100:00100000:3.0:1789760722.037054:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484179584 00000100:00100000:3.0:1789760722.037206:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711269023554 time 711323591617 state 0x0 ref 1 00000100:00100000:3.0:1789760722.037226:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711323852033 time 711323591617 state 0x1 ref 1 00000100:00100000:3.0:1789760722.037859:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711323852033 time 711368784666 state 0x0 ref 1 00000100:00100000:3.0:1789760722.037880:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484179584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760722.037947:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19755008, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:3.0:1789760722.037958:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.037977:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:3.0:1789760722.037982:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760722.037989:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19755008->19759103] (req 19755008->19759103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.038005:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19755008->19759103] (req 19755008->19759103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1789760722.040153:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:3.0:1789760722.110283:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19755008 -> 19759104 00010000:00010000:3.0:1789760722.110314:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19755008->19759103] (req 19755008->19759103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.110329:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19755008->19759103] (req 19755008->19759103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.110339:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19755008->19759103] (req 19755008->19759103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.110346:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760722.110352:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19755008->19759103] (req 19755008->19759103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.110363:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19755008->19759103] (req 19755008->19759103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.110392:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19755008->19759103] (req 19755008->19759103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.110402:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207d9e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19755008->19759103] (req 19755008->19759103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760722.110553:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484179584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 72664us (73622us total) trans 4294972141 rc 0/0 00000100:00100000:3.0:1789760722.110568:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4852 00000100:00100000:3.0:1789760722.110582:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711323852033 time 711368784666 state 0x2 ref 0 00000001:00080000:2.0:1789760722.111663:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972141 is committed 00000100:00100000:3.0:1789760722.115059:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760722.116019:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484179712 00000100:00100000:1.0:1789760722.116033:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484179712 00000100:00100000:1.0:1789760722.116347:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711323852033 time 711368784666 state 0x0 ref 1 00000100:00100000:1.0:1789760722.116368:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711369045082 time 711368784666 state 0x1 ref 1 00000100:00100000:2.0:1789760722.116530:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711369045082 time 711447455971 state 0x0 ref 1 00000100:00100000:2.0:1789760722.116618:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484179712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760722.116688:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19759104, mtime: 0x6aad94d1, atime: 0x0, ctime: 0x6aad94d1, blocks: 0x9807 00010000:00010000:2.0:1789760722.116699:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.116719:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760722.116723:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760722.116730:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19759104->19763199] (req 19759104->19763199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.116745:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19759104->19763199] (req 19759104->19763199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760722.151817:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760722.151869:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760722.151891:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760722.151915:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760722.151939:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760722.151971:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484179968 00000100:00100000:1.0:1789760722.151985:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484179968 00000100:00100000:1.0:1789760722.152094:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484180096 00000100:00100000:1.0:1789760722.152098:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484180096 00000100:00100000:1.0:1789760722.152122:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484180352 00000100:00100000:1.0:1789760722.152125:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484180352 00000100:00100000:1.0:1789760722.152138:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484180224 00000100:00100000:1.0:1789760722.152140:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484180224 00000100:00100000:1.0:1789760722.152158:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 881 00000100:00100000:1.0:1789760722.152169:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484179968:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760722.152264:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484179968:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 96us (467us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760722.152272:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 881 00000100:00100000:1.0:1789760722.152290:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 882 00000100:00100000:1.0:1789760722.152296:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82d500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484180096:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760722.152325:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82d500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484180096:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 28us (433us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760722.152331:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 882 00000100:00100000:1.0:1789760722.152341:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 883 00000100:00100000:1.0:1789760722.152347:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484180352:12345-192.168.203.10@tcp:400:kworker.0 00000100:00080000:3.0:1789760722.153122:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:3.0:1789760722.153128:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:1.0:1789760722.156163:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484180352:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 3799us (4229us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760722.156181:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 883 00000100:00100000:1.0:1789760722.156232:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 884 00000100:00100000:1.0:1789760722.156251:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484180224:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760722.156283:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484180224:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 41us (4343us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760722.156291:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 884 00000100:00100000:1.0:1789760722.156327:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484179840 00000100:00100000:1.0:1789760722.156337:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484179840 00000100:00100000:1.0:1789760722.156781:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 471 00000100:00100000:1.0:1789760722.156791:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b6850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484179840:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760722.156824:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b6850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484179840:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 35us (4954us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760722.156831:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 471 00000001:00080000:2.0:1789760722.171760:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972142 is committed 00002000:00010000:3.0:1789760722.171965:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19759104 -> 19763200 00002000:00010000:3.0:1789760722.171971:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760721 -> 1789760722 00002000:00010000:3.0:1789760722.171974:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760721 -> 1789760722 00010000:00010000:3.0:1789760722.171992:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19759104->19763199] (req 19759104->19763199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.172008:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19759104->19763199] (req 19759104->19763199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.172019:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19759104->19763199] (req 19759104->19763199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.172027:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760722.172033:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19759104->19763199] (req 19759104->19763199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.172043:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19759104->19763199] (req 19759104->19763199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.172117:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19759104->19763199] (req 19759104->19763199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.172127:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207da5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19759104->19763199] (req 19759104->19763199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760722.172680:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484179712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55702us (57202us total) trans 4294972142 rc 0/0 00000100:00100000:3.0:1789760722.172693:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4853 00000100:00100000:3.0:1789760722.172705:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711369045082 time 711447455971 state 0x2 ref 0 00000100:00100000:2.0:1789760722.175888:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760722.176008:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484180480 00000100:00100000:0.0:1789760722.176021:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484180480 00000100:00100000:0.0:1789760722.176144:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711369045082 time 711447455971 state 0x0 ref 1 00000100:00100000:0.0:1789760722.176165:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711447716387 time 711447455971 state 0x1 ref 1 00000100:00100000:0.0:1789760722.176196:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711447716387 time 711507125924 state 0x0 ref 1 00000100:00100000:0.0:1789760722.176213:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484180480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760722.176391:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19763200, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:0.0:1789760722.176404:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.176426:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:0.0:1789760722.176431:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760722.176444:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19763200->19767295] (req 19763200->19767295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.176470:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19763200->19767295] (req 19763200->19767295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760722.209775:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19763200 -> 19767296 00010000:00010000:0.0:1789760722.209807:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19763200->19767295] (req 19763200->19767295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.209824:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19763200->19767295] (req 19763200->19767295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.209834:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19763200->19767295] (req 19763200->19767295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.209842:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760722.209848:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19763200->19767295] (req 19763200->19767295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.209858:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19763200->19767295] (req 19763200->19767295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.209887:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19763200->19767295] (req 19763200->19767295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.209895:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207dac lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19763200->19767295] (req 19763200->19767295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760722.210106:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484180480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33858us (34191us total) trans 4294972143 rc 0/0 00000100:00100000:0.0:1789760722.210119:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4854 00000100:00100000:0.0:1789760722.210131:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711447716387 time 711507125924 state 0x2 ref 0 00000001:00080000:3.0:1789760722.210426:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972143 is committed 00000100:00100000:1.0:1789760722.215510:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760722.215654:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484180608 00000100:00100000:2.0:1789760722.215669:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484180608 00000100:00100000:2.0:1789760722.215749:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711447716387 time 711507125924 state 0x0 ref 1 00000100:00100000:2.0:1789760722.215769:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711507386340 time 711507125924 state 0x1 ref 1 00000100:00100000:2.0:1789760722.215859:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711507386340 time 711546787901 state 0x0 ref 1 00000100:00100000:2.0:1789760722.215876:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484180608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760722.215944:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19767296, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:2.0:1789760722.215955:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.215975:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760722.215980:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760722.215988:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19767296->19771391] (req 19767296->19771391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.216005:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19767296->19771391] (req 19767296->19771391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760722.247986:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972144 is committed 00002000:00010000:3.0:1789760722.248062:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19767296 -> 19771392 00010000:00010000:3.0:1789760722.248103:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19767296->19771391] (req 19767296->19771391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.248116:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19767296->19771391] (req 19767296->19771391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.248123:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19767296->19771391] (req 19767296->19771391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.248128:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760722.248133:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19767296->19771391] (req 19767296->19771391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.248140:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19767296->19771391] (req 19767296->19771391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.248161:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19767296->19771391] (req 19767296->19771391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.248167:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207db3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19767296->19771391] (req 19767296->19771391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760722.248270:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484180608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32392us (32768us total) trans 4294972144 rc 0/0 00000100:00100000:3.0:1789760722.248281:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4855 00000100:00100000:3.0:1789760722.248294:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711507386340 time 711546787901 state 0x2 ref 0 00000100:00100000:2.0:1789760722.251753:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760722.251931:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484180736 00000100:00100000:3.0:1789760722.251944:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484180736 00000100:00100000:3.0:1789760722.252021:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711507386340 time 711546787901 state 0x0 ref 1 00000100:00100000:3.0:1789760722.252041:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711547048317 time 711546787901 state 0x1 ref 1 00000100:00100000:0.0:1789760722.252142:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711547048317 time 711583066988 state 0x0 ref 1 00000100:00100000:2.0:1789760722.252229:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484180736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760722.252296:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19771392, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:2.0:1789760722.252304:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.252325:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760722.252329:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760722.252335:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19771392->19775487] (req 19771392->19775487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.252347:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19771392->19775487] (req 19771392->19775487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760722.286544:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972145 is committed 00002000:00010000:3.0:1789760722.286799:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19771392 -> 19775488 00010000:00010000:3.0:1789760722.286819:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19771392->19775487] (req 19771392->19775487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.286835:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19771392->19775487] (req 19771392->19775487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.286845:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19771392->19775487] (req 19771392->19775487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.286854:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760722.286860:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19771392->19775487] (req 19771392->19775487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.286871:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19771392->19775487] (req 19771392->19775487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.286898:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19771392->19775487] (req 19771392->19775487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.286906:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207dba lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19771392->19775487] (req 19771392->19775487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760722.287062:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484180736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34899us (35378us total) trans 4294972145 rc 0/0 00000100:00100000:3.0:1789760722.287115:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4856 00000100:00100000:3.0:1789760722.287128:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711547048317 time 711583066988 state 0x2 ref 0 00000100:00100000:1.0:1789760722.291308:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760722.291417:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484180864 00000100:00100000:3.0:1789760722.291430:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484180864 00000100:00100000:3.0:1789760722.291539:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711547048317 time 711583066988 state 0x0 ref 1 00000100:00100000:3.0:1789760722.291624:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711583327404 time 711583066988 state 0x1 ref 1 00000100:00100000:3.0:1789760722.291778:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711583327404 time 711622706699 state 0x0 ref 1 00000100:00100000:3.0:1789760722.291795:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484180864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760722.291860:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19775488, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:3.0:1789760722.291872:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.291892:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:3.0:1789760722.291898:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760722.291906:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19775488->19779583] (req 19775488->19779583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.291923:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19775488->19779583] (req 19775488->19779583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760722.330224:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972146 is committed 00002000:00010000:0.0:1789760722.330381:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19775488 -> 19779584 00010000:00010000:0.0:1789760722.330403:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19775488->19779583] (req 19775488->19779583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.330417:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19775488->19779583] (req 19775488->19779583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.330437:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19775488->19779583] (req 19775488->19779583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.330446:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760722.330452:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19775488->19779583] (req 19775488->19779583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.330496:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19775488->19779583] (req 19775488->19779583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.330522:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19775488->19779583] (req 19775488->19779583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.330530:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207dc1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19775488->19779583] (req 19775488->19779583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760722.330670:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484180864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38872us (39428us total) trans 4294972146 rc 0/0 00000100:00100000:0.0:1789760722.330683:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4857 00000100:00100000:0.0:1789760722.330695:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711583327404 time 711622706699 state 0x2 ref 0 00000100:00100000:1.0:1789760722.337968:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760722.338135:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484180992 00000100:00100000:2.0:1789760722.338204:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484180992 00000100:00100000:2.0:1789760722.338300:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711583327404 time 711622706699 state 0x0 ref 1 00000100:00100000:2.0:1789760722.338320:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711622967115 time 711622706699 state 0x1 ref 1 00000100:00100000:2.0:1789760722.338354:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711622967115 time 711669285264 state 0x0 ref 1 00000100:00100000:2.0:1789760722.338370:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484180992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760722.338437:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19779584, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:2.0:1789760722.338448:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.338479:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760722.338485:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760722.338491:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19779584->19783679] (req 19779584->19783679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.338512:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19779584->19783679] (req 19779584->19783679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760722.383479:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972147 is committed 00002000:00010000:2.0:1789760722.383886:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19779584 -> 19783680 00010000:00010000:2.0:1789760722.383907:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19779584->19783679] (req 19779584->19783679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.383921:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19779584->19783679] (req 19779584->19783679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.383930:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19779584->19783679] (req 19779584->19783679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.383937:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760722.383943:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19779584->19783679] (req 19779584->19783679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.383953:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19779584->19783679] (req 19779584->19783679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.383980:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19779584->19783679] (req 19779584->19783679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.383989:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dc8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19779584->19783679] (req 19779584->19783679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760722.384344:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484180992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45969us (46384us total) trans 4294972147 rc 0/0 00000100:00100000:2.0:1789760722.384357:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4858 00000100:00100000:2.0:1789760722.384370:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711622967115 time 711669285264 state 0x2 ref 0 00000100:00100000:1.0:1789760722.392238:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760722.392376:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484181120 00000100:00100000:2.0:1789760722.392392:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484181120 00000100:00100000:2.0:1789760722.393002:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711622967115 time 711669285264 state 0x0 ref 1 00000100:00100000:2.0:1789760722.393026:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711669545680 time 711669285264 state 0x1 ref 1 00000100:00100000:2.0:1789760722.393141:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711669545680 time 711724070499 state 0x0 ref 1 00000100:00100000:2.0:1789760722.393160:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484181120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760722.393227:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19783680, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:2.0:1789760722.393238:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.393257:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760722.393262:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760722.393269:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19783680->19787775] (req 19783680->19787775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.393285:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19783680->19787775] (req 19783680->19787775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760722.443314:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972148 is committed 00002000:00010000:2.0:1789760722.443702:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19783680 -> 19787776 00010000:00010000:2.0:1789760722.443723:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19783680->19787775] (req 19783680->19787775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.443800:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19783680->19787775] (req 19783680->19787775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.443811:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19783680->19787775] (req 19783680->19787775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.443819:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760722.443825:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19783680->19787775] (req 19783680->19787775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.443836:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19783680->19787775] (req 19783680->19787775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.443861:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19783680->19787775] (req 19783680->19787775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.443869:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207dcf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19783680->19787775] (req 19783680->19787775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760722.444187:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484181120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50878us (51811us total) trans 4294972148 rc 0/0 00000100:00100000:2.0:1789760722.444205:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4859 00000100:00100000:2.0:1789760722.444218:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711669545680 time 711724070499 state 0x2 ref 0 00000100:00100000:1.0:1789760722.451211:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760722.451317:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484181248 00000100:00100000:3.0:1789760722.451334:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484181248 00000100:00100000:3.0:1789760722.451415:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711669545680 time 711724070499 state 0x0 ref 1 00000100:00100000:3.0:1789760722.451444:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711724330915 time 711724070499 state 0x1 ref 1 00000100:00100000:3.0:1789760722.451481:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711724330915 time 711782411458 state 0x0 ref 1 00000100:00100000:3.0:1789760722.451496:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484181248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760722.451561:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19787776, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:3.0:1789760722.451572:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.451593:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:3.0:1789760722.451599:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760722.451607:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19787776->19791871] (req 19787776->19791871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.451636:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19787776->19791871] (req 19787776->19791871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760722.521176:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972149 is committed 00002000:00010000:0.0:1789760722.524036:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19787776 -> 19791872 00010000:00010000:0.0:1789760722.524191:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19787776->19791871] (req 19787776->19791871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.524206:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19787776->19791871] (req 19787776->19791871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.524216:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19787776->19791871] (req 19787776->19791871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.524224:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760722.524231:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19787776->19791871] (req 19787776->19791871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.524241:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19787776->19791871] (req 19787776->19791871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.524368:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19787776->19791871] (req 19787776->19791871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.524375:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207dd6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19787776->19791871] (req 19787776->19791871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760722.524666:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484181248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73166us (73467us total) trans 4294972149 rc 0/0 00000100:00100000:0.0:1789760722.524679:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4860 00000100:00100000:0.0:1789760722.524692:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711724330915 time 711782411458 state 0x2 ref 0 00000100:00100000:1.0:1789760722.531230:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760722.531398:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484181376 00000100:00100000:0.0:1789760722.531414:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484181376 00000100:00100000:0.0:1789760722.531510:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711724330915 time 711782411458 state 0x0 ref 1 00000100:00100000:0.0:1789760722.531528:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711782671874 time 711782411458 state 0x1 ref 1 00000100:00100000:0.0:1789760722.531564:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711782671874 time 711862495019 state 0x0 ref 1 00000100:00100000:0.0:1789760722.531582:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484181376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760722.531646:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19791872, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:0.0:1789760722.531659:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.531682:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:0.0:1789760722.531687:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760722.531695:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19791872->19795967] (req 19791872->19795967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.531710:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19791872->19795967] (req 19791872->19795967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760722.593082:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972150 is committed 00002000:00010000:3.0:1789760722.593295:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19791872 -> 19795968 00010000:00010000:3.0:1789760722.593315:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19791872->19795967] (req 19791872->19795967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.593330:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19791872->19795967] (req 19791872->19795967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.593340:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19791872->19795967] (req 19791872->19795967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.593349:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760722.593354:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19791872->19795967] (req 19791872->19795967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.593364:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19791872->19795967] (req 19791872->19795967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.593390:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19791872->19795967] (req 19791872->19795967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.593398:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396207ddd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19791872->19795967] (req 19791872->19795967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760722.593668:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484181376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62084us (62448us total) trans 4294972150 rc 0/0 00000100:00100000:3.0:1789760722.593682:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4861 00000100:00100000:3.0:1789760722.593694:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711782671874 time 711862495019 state 0x2 ref 0 00000100:00100000:1.0:1789760722.599292:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760722.599508:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484181504 00000100:00100000:2.0:1789760722.599521:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484181504 00000100:00100000:2.0:1789760722.599663:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711782671874 time 711862495019 state 0x0 ref 1 00000100:00100000:2.0:1789760722.599681:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711862755435 time 711862495019 state 0x1 ref 1 00000100:00100000:2.0:1789760722.599719:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711862755435 time 711930649973 state 0x0 ref 1 00000100:00100000:2.0:1789760722.599733:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484181504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760722.599894:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19795968, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:2.0:1789760722.599904:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.599922:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=711s v=(0 0) 00010000:00010000:2.0:1789760722.599926:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760722.599934:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19795968->19800063] (req 19795968->19800063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.599950:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19795968->19800063] (req 19795968->19800063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760722.670732:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972151 is committed 00002000:00010000:3.0:1789760722.670868:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19795968 -> 19800064 00010000:00010000:3.0:1789760722.670889:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19795968->19800063] (req 19795968->19800063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.670904:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19795968->19800063] (req 19795968->19800063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.670914:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19795968->19800063] (req 19795968->19800063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.670923:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760722.670928:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19795968->19800063] (req 19795968->19800063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.670939:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19795968->19800063] (req 19795968->19800063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.670967:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19795968->19800063] (req 19795968->19800063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.670982:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207de4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19795968->19800063] (req 19795968->19800063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760722.671153:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484181504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 71415us (71866us total) trans 4294972151 rc 0/0 00000100:00100000:3.0:1789760722.671165:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4862 00000100:00100000:3.0:1789760722.671177:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711862755435 time 711930649973 state 0x2 ref 0 00000100:00100000:1.0:1789760722.674633:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760722.674734:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484181632 00000100:00100000:2.0:1789760722.674749:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484181632 00000100:00100000:2.0:1789760722.674823:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711862755435 time 711930649973 state 0x0 ref 1 00000100:00100000:2.0:1789760722.674841:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711930910389 time 711930649973 state 0x1 ref 1 00000100:00100000:2.0:1789760722.674877:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711930910389 time 712005807675 state 0x0 ref 1 00000100:00100000:2.0:1789760722.674892:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484181632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760722.674977:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19800064, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:2.0:1789760722.674989:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.675008:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:2.0:1789760722.675014:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760722.675022:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19800064->19804159] (req 19800064->19804159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.675039:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19800064->19804159] (req 19800064->19804159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760722.713697:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972152 is committed 00002000:00010000:3.0:1789760722.713835:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19800064 -> 19804160 00010000:00010000:3.0:1789760722.713857:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19800064->19804159] (req 19800064->19804159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.713875:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19800064->19804159] (req 19800064->19804159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.713886:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19800064->19804159] (req 19800064->19804159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.713895:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760722.713901:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19800064->19804159] (req 19800064->19804159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.713913:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19800064->19804159] (req 19800064->19804159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.713944:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19800064->19804159] (req 19800064->19804159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.713953:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207deb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19800064->19804159] (req 19800064->19804159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760722.714954:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484181632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40053us (40326us total) trans 4294972152 rc 0/0 00000100:00100000:3.0:1789760722.714969:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4863 00000100:00100000:3.0:1789760722.714982:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711930910389 time 712005807675 state 0x2 ref 0 00000100:00100000:0.0:1789760722.718247:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760722.718356:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484181760 00000100:00100000:3.0:1789760722.718371:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484181760 00000100:00100000:3.0:1789760722.718443:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 711930910389 time 712005807675 state 0x0 ref 1 00000100:00100000:3.0:1789760722.718476:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712006068091 time 712005807675 state 0x1 ref 1 00000100:00100000:3.0:1789760722.718510:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712006068091 time 712049440283 state 0x0 ref 1 00000100:00100000:3.0:1789760722.718524:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484181760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760722.718589:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19804160, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:3.0:1789760722.718600:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.718621:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:3.0:1789760722.718628:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760722.718636:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19804160->19808255] (req 19804160->19808255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.718653:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19804160->19808255] (req 19804160->19808255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760722.753619:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972153 is committed 00002000:00010000:3.0:1789760722.753733:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19804160 -> 19808256 00010000:00010000:3.0:1789760722.753753:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19804160->19808255] (req 19804160->19808255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.753769:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19804160->19808255] (req 19804160->19808255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.753780:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19804160->19808255] (req 19804160->19808255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.753788:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760722.753794:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19804160->19808255] (req 19804160->19808255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.753804:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19804160->19808255] (req 19804160->19808255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.753829:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19804160->19808255] (req 19804160->19808255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.753838:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207df2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19804160->19808255] (req 19804160->19808255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760722.754191:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484181760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35662us (35952us total) trans 4294972153 rc 0/0 00000100:00100000:3.0:1789760722.754203:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4864 00000100:00100000:3.0:1789760722.754217:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712006068091 time 712049440283 state 0x2 ref 0 00000100:00100000:2.0:1789760722.757920:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760722.758737:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484181888 00000100:00100000:3.0:1789760722.758753:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484181888 00000100:00100000:2.0:1789760722.758916:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712006068091 time 712049440283 state 0x0 ref 1 00000100:00100000:2.0:1789760722.758934:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712049700699 time 712049440283 state 0x1 ref 1 00000100:00100000:2.0:1789760722.758972:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712049700699 time 712089902027 state 0x0 ref 1 00000100:00100000:2.0:1789760722.758988:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484181888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760722.759053:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19808256, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:2.0:1789760722.759140:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.759160:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:2.0:1789760722.759165:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760722.759172:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19808256->19812351] (req 19808256->19812351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.759189:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19808256->19812351] (req 19808256->19812351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760722.807124:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972154 is committed 00002000:00010000:0.0:1789760722.807221:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19808256 -> 19812352 00010000:00010000:0.0:1789760722.807248:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19808256->19812351] (req 19808256->19812351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.807264:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19808256->19812351] (req 19808256->19812351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.807274:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19808256->19812351] (req 19808256->19812351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.807282:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760722.807288:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19808256->19812351] (req 19808256->19812351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.807298:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19808256->19812351] (req 19808256->19812351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.807322:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19808256->19812351] (req 19808256->19812351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.807330:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207df9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19808256->19812351] (req 19808256->19812351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760722.807443:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484181888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48453us (49534us total) trans 4294972154 rc 0/0 00000100:00100000:0.0:1789760722.807469:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4865 00000100:00100000:0.0:1789760722.807648:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712049700699 time 712089902027 state 0x2 ref 0 00000100:00100000:1.0:1789760722.811679:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760722.811787:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484182016 00000100:00100000:0.0:1789760722.811801:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484182016 00000100:00100000:0.0:1789760722.811883:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712049700699 time 712089902027 state 0x0 ref 1 00000100:00100000:0.0:1789760722.811902:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712090162443 time 712089902027 state 0x1 ref 1 00000100:00100000:0.0:1789760722.811938:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712090162443 time 712142868273 state 0x0 ref 1 00000100:00100000:0.0:1789760722.811954:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760722.812022:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19812352, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:0.0:1789760722.812034:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.812055:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:0.0:1789760722.812061:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760722.812191:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19812352->19816447] (req 19812352->19816447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.812208:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19812352->19816447] (req 19812352->19816447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760722.828200:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760722.828246:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508912896:0@lo:41:osp-pre-0-1.0 00000100:00100000:1.0:1789760722.828338:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760722.828431:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508912896 00000100:00100000:1.0:1789760722.828442:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508912896 00000100:00100000:1.0:1789760722.828509:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 425 00000100:00100000:1.0:1789760722.828521:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508912896:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:1.0:1789760722.828572:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:1.0:1789760722.828633:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018adf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508912896:12345-0@lo:41:osp-pre-0-1.0 Request processed in 108us (289us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760722.828641:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 425 00000100:00100000:1.0:1789760722.828702:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508912896:0@lo:41:osp-pre-0-1.0 00010000:00010000:1.0:1789760722.829215:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760722.829322:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760722.829334:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508913024:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760722.829608:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760722.829666:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8109f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508913152:0@lo:13:osp-pre-0-1.0 00000100:00100000:3.0:1789760722.829687:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760722.829721:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508913024 00000100:00100000:3.0:1789760722.829726:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508913024 00000100:00100000:3.0:1789760722.829759:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508913152 00000100:00100000:3.0:1789760722.829761:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508913152 00000100:00100000:3.0:1789760722.829770:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 366 00000100:00100000:3.0:1789760722.829782:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508913024:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:3.0:1789760722.829800:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527892480 pending 1703936 free 3838836736 avail 3814719488 00000100:00100000:0.0:1789760722.829821:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 367 00000100:00100000:3.0:1789760722.829832:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508913024:12345-0@lo:13:osp-pre-0-0.0 Request processed in 55us (450us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760722.829837:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 366 00000100:00100000:0.0:1789760722.829844:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508913152:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:0.0:1789760722.829858:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527892480 pending 1703936 free 3838836736 avail 3814719488 00000100:00100000:3.0:1789760722.829865:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508913024:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760722.829962:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508913152:12345-0@lo:13:osp-pre-0-1.0 Request processed in 131us (271us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760722.829969:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 367 00000100:00100000:3.0:1789760722.831019:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8109f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508913152:0@lo:13:osp-pre-0-1.0 00000001:00080000:1.0:1789760722.853551:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972155 is committed 00002000:00010000:0.0:1789760722.853756:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19812352 -> 19816448 00010000:00010000:0.0:1789760722.853787:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19812352->19816447] (req 19812352->19816447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.853802:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19812352->19816447] (req 19812352->19816447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.853811:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19812352->19816447] (req 19812352->19816447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.853819:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760722.853825:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19812352->19816447] (req 19812352->19816447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.853842:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19812352->19816447] (req 19812352->19816447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.853868:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19812352->19816447] (req 19812352->19816447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.853875:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e00 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19812352->19816447] (req 19812352->19816447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760722.853987:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42030us (42318us total) trans 4294972155 rc 0/0 00000100:00100000:0.0:1789760722.854000:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4866 00000100:00100000:0.0:1789760722.854013:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712090162443 time 712142868273 state 0x2 ref 0 00000100:00100000:1.0:1789760722.856819:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760722.857261:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484182144 00000100:00100000:0.0:1789760722.857275:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484182144 00000100:00100000:0.0:1789760722.857358:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712090162443 time 712142868273 state 0x0 ref 1 00000100:00100000:0.0:1789760722.857377:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712143128689 time 712142868273 state 0x1 ref 1 00000100:00100000:0.0:1789760722.857416:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712143128689 time 712188346232 state 0x0 ref 1 00000100:00100000:0.0:1789760722.857432:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760722.857514:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19816448, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:0.0:1789760722.857525:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.857547:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:0.0:1789760722.857552:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760722.857559:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19816448->19820543] (req 19816448->19820543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.857577:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19816448->19820543] (req 19816448->19820543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760722.915889:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972156 is committed 00002000:00010000:2.0:1789760722.916184:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19816448 -> 19820544 00010000:00010000:2.0:1789760722.916204:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19816448->19820543] (req 19816448->19820543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.916220:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19816448->19820543] (req 19816448->19820543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.916230:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19816448->19820543] (req 19816448->19820543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.916303:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760722.916308:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19816448->19820543] (req 19816448->19820543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.916322:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19816448->19820543] (req 19816448->19820543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.916350:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19816448->19820543] (req 19816448->19820543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.916359:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207e07 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19816448->19820543] (req 19816448->19820543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760722.916538:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59103us (59726us total) trans 4294972156 rc 0/0 00000100:00100000:2.0:1789760722.916616:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4867 00000100:00100000:2.0:1789760722.916629:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712143128689 time 712188346232 state 0x2 ref 0 00000100:00100000:2.0:1789760722.918993:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760722.919204:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484182272 00000100:00100000:3.0:1789760722.919223:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484182272 00000100:00100000:3.0:1789760722.919365:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712143128689 time 712188346232 state 0x0 ref 1 00000100:00100000:3.0:1789760722.919385:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712188606648 time 712188346232 state 0x1 ref 1 00000100:00100000:3.0:1789760722.919470:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712188606648 time 712250399501 state 0x0 ref 1 00000100:00100000:3.0:1789760722.919487:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484182272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760722.919615:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19820544, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:3.0:1789760722.919629:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.919652:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:3.0:1789760722.919658:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760722.919667:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19820544->19824639] (req 19820544->19824639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760722.919686:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19820544->19824639] (req 19820544->19824639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760722.989991:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972157 is committed 00002000:00010000:0.0:1789760722.990144:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19820544 -> 19824640 00010000:00010000:0.0:1789760722.990165:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19820544->19824639] (req 19820544->19824639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.990181:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19820544->19824639] (req 19820544->19824639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.990191:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19820544->19824639] (req 19820544->19824639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.990199:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760722.990204:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19820544->19824639] (req 19820544->19824639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.990215:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19820544->19824639] (req 19820544->19824639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.990242:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19820544->19824639] (req 19820544->19824639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760722.990251:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e0e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19820544->19824639] (req 19820544->19824639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760722.990394:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484182272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 70903us (71409us total) trans 4294972157 rc 0/0 00000100:00100000:0.0:1789760722.990407:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4868 00000100:00100000:0.0:1789760722.990420:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712188606648 time 712250399501 state 0x2 ref 0 00000100:00100000:1.0:1789760722.993240:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760722.993461:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484182400 00000100:00100000:2.0:1789760722.993474:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484182400 00000100:00100000:2.0:1789760722.993547:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712188606648 time 712250399501 state 0x0 ref 1 00000100:00100000:2.0:1789760722.993566:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712250659917 time 712250399501 state 0x1 ref 1 00000100:00100000:2.0:1789760722.993599:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712250659917 time 712324529417 state 0x0 ref 1 00000100:00100000:2.0:1789760722.993614:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760722.993676:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19824640, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:2.0:1789760722.993687:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.993706:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:2.0:1789760722.993711:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760722.993718:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19824640->19828735] (req 19824640->19828735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760722.993734:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19824640->19828735] (req 19824640->19828735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760723.067742:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972158 is committed 00002000:00010000:0.0:1789760723.067789:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19824640 -> 19828736 00010000:00010000:0.0:1789760723.067815:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19824640->19828735] (req 19824640->19828735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.067831:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19824640->19828735] (req 19824640->19828735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.067842:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19824640->19828735] (req 19824640->19828735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.067851:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760723.067856:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19824640->19828735] (req 19824640->19828735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.067866:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19824640->19828735] (req 19824640->19828735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.067895:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19824640->19828735] (req 19824640->19828735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.067904:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396207e15 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19824640->19828735] (req 19824640->19828735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760723.068037:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 74418us (74804us total) trans 4294972158 rc 0/0 00000100:00100000:0.0:1789760723.068049:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4869 00000100:00100000:0.0:1789760723.068106:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712250659917 time 712324529417 state 0x2 ref 0 00000100:00100000:1.0:1789760723.072397:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760723.072527:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484182528 00000100:00100000:0.0:1789760723.072541:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484182528 00000100:00100000:0.0:1789760723.072636:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712250659917 time 712324529417 state 0x0 ref 1 00000100:00100000:0.0:1789760723.072656:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712324789833 time 712324529417 state 0x1 ref 1 00000100:00100000:0.0:1789760723.072690:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712324789833 time 712403621550 state 0x0 ref 1 00000100:00100000:0.0:1789760723.072708:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7073450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760723.072771:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19828736, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:0.0:1789760723.072791:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.072813:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:0.0:1789760723.072820:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760723.072828:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19828736->19832831] (req 19828736->19832831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.072846:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19828736->19832831] (req 19828736->19832831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760723.149773:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19828736 -> 19832832 00010000:00010000:3.0:1789760723.149802:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19828736->19832831] (req 19828736->19832831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.149817:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19828736->19832831] (req 19828736->19832831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.149829:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19828736->19832831] (req 19828736->19832831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.149838:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760723.149844:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19828736->19832831] (req 19828736->19832831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.149855:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19828736->19832831] (req 19828736->19832831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.149881:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19828736->19832831] (req 19828736->19832831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.149890:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207e1c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19828736->19832831] (req 19828736->19832831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760723.152671:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7073450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484182528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 78519us (78840us total) trans 4294972159 rc 0/0 00000100:00100000:3.0:1789760723.152692:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4870 00000100:00100000:3.0:1789760723.152705:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712324789833 time 712403621550 state 0x2 ref 0 00000001:00080000:3.0:1789760723.152964:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972159 is committed 00000100:00100000:2.0:1789760723.163878:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760723.163973:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484182656 00000100:00100000:0.0:1789760723.163991:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484182656 00000100:00100000:0.0:1789760723.164139:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712324789833 time 712403621550 state 0x0 ref 1 00000100:00100000:0.0:1789760723.164161:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712403881966 time 712403621550 state 0x1 ref 1 00000100:00100000:0.0:1789760723.164534:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712403881966 time 712495462235 state 0x0 ref 1 00000100:00100000:0.0:1789760723.164557:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760723.164630:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19832832, mtime: 0x6aad94d2, atime: 0x0, ctime: 0x6aad94d2, blocks: 0x9807 00010000:00010000:0.0:1789760723.164642:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.164664:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:0.0:1789760723.164669:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760723.164677:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19832832->19836927] (req 19832832->19836927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.164694:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19832832->19836927] (req 19832832->19836927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760723.249761:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972160 is committed 00002000:00010000:2.0:1789760723.249956:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19832832 -> 19836928 00002000:00010000:2.0:1789760723.249962:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760722 -> 1789760723 00002000:00010000:2.0:1789760723.249965:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760722 -> 1789760723 00010000:00010000:2.0:1789760723.249983:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19832832->19836927] (req 19832832->19836927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.249998:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19832832->19836927] (req 19832832->19836927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.250008:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19832832->19836927] (req 19832832->19836927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.250016:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760723.250020:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19832832->19836927] (req 19832832->19836927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.250035:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19832832->19836927] (req 19832832->19836927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.250060:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19832832->19836927] (req 19832832->19836927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.250203:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396207e23 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19832832->19836927] (req 19832832->19836927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760723.250609:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 86053us (86733us total) trans 4294972160 rc 0/0 00000100:00100000:3.0:1789760723.250623:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4871 00000100:00100000:3.0:1789760723.250636:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712403881966 time 712495462235 state 0x2 ref 0 00000100:00100000:1.0:1789760723.255817:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760723.256022:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484182784 00000100:00100000:3.0:1789760723.256035:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484182784 00000100:00100000:3.0:1789760723.256154:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712403881966 time 712495462235 state 0x0 ref 1 00000100:00100000:3.0:1789760723.256175:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712495722651 time 712495462235 state 0x1 ref 1 00000100:00100000:3.0:1789760723.256208:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712495722651 time 712587138952 state 0x0 ref 1 00000100:00100000:3.0:1789760723.256224:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760723.256594:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19836928, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:3.0:1789760723.256607:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.256632:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:3.0:1789760723.256638:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760723.256646:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19836928->19841023] (req 19836928->19841023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.256665:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19836928->19841023] (req 19836928->19841023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760723.316360:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19836928 -> 19841024 00010000:00010000:3.0:1789760723.316387:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19836928->19841023] (req 19836928->19841023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.316409:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19836928->19841023] (req 19836928->19841023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.316419:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19836928->19841023] (req 19836928->19841023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.316426:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760723.316432:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19836928->19841023] (req 19836928->19841023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.316442:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19836928->19841023] (req 19836928->19841023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.316478:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19836928->19841023] (req 19836928->19841023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.316487:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e2a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19836928->19841023] (req 19836928->19841023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760723.316618:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484182784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60390us (60810us total) trans 4294972161 rc 0/0 00000100:00100000:3.0:1789760723.316632:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4872 00000100:00100000:3.0:1789760723.316645:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712495722651 time 712587138952 state 0x2 ref 0 00000001:00080000:1.0:1789760723.318014:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972161 is committed 00000100:00100000:1.0:1789760723.320281:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760723.321221:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484182912 00000100:00100000:3.0:1789760723.321238:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484182912 00000100:00100000:3.0:1789760723.321317:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712495722651 time 712587138952 state 0x0 ref 1 00000100:00100000:3.0:1789760723.321337:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712587399368 time 712587138952 state 0x1 ref 1 00000100:00100000:3.0:1789760723.321436:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712587399368 time 712652365919 state 0x0 ref 1 00000100:00100000:3.0:1789760723.321460:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760723.321528:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19841024, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:3.0:1789760723.321538:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.321559:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:3.0:1789760723.321564:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760723.321571:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19841024->19845119] (req 19841024->19845119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.321589:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19841024->19845119] (req 19841024->19845119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760723.342992:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508913280:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760723.343107:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760723.343355:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508913280 00000100:00100000:1.0:1789760723.343366:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508913280 00000100:00100000:1.0:1789760723.343425:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 426 00000100:00100000:1.0:1789760723.343437:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508913280:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760723.343503:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760723.343558:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508913280:12345-0@lo:41:osp-pre-1-0.0 Request processed in 122us (451us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760723.343566:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 426 00000100:00100000:1.0:1789760723.343626:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508913280:0@lo:41:osp-pre-1-0.0 00002000:00010000:3.0:1789760723.387913:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19841024 -> 19845120 00010000:00010000:3.0:1789760723.387946:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19841024->19845119] (req 19841024->19845119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.387964:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19841024->19845119] (req 19841024->19845119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.387980:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19841024->19845119] (req 19841024->19845119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.387988:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760723.387994:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19841024->19845119] (req 19841024->19845119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.388005:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19841024->19845119] (req 19841024->19845119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.388030:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19841024->19845119] (req 19841024->19845119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.388040:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e31 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19841024->19845119] (req 19841024->19845119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760723.388177:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972162 is committed 00000100:00100000:2.0:1789760723.388683:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484182912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67226us (68407us total) trans 4294972162 rc 0/0 00000100:00100000:2.0:1789760723.388696:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4873 00000100:00100000:2.0:1789760723.388710:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712587399368 time 712652365919 state 0x2 ref 0 00000100:00100000:2.0:1789760723.391813:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760723.392266:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484183040 00000100:00100000:0.0:1789760723.392281:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484183040 00000100:00100000:2.0:1789760723.392486:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712587399368 time 712652365919 state 0x0 ref 1 00000100:00100000:2.0:1789760723.392505:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712652626335 time 712652365919 state 0x1 ref 1 00000100:00100000:2.0:1789760723.392636:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712652626335 time 712723564678 state 0x0 ref 1 00000100:00100000:2.0:1789760723.392660:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484183040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760723.392729:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19845120, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:2.0:1789760723.392739:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.392760:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:2.0:1789760723.392764:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760723.392772:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19845120->19849215] (req 19845120->19849215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.392788:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19845120->19849215] (req 19845120->19849215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760723.444647:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972163 is committed 00002000:00010000:0.0:1789760723.444869:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19845120 -> 19849216 00010000:00010000:0.0:1789760723.444890:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19845120->19849215] (req 19845120->19849215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.444904:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19845120->19849215] (req 19845120->19849215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.444914:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19845120->19849215] (req 19845120->19849215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.444923:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760723.444929:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19845120->19849215] (req 19845120->19849215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.444940:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19845120->19849215] (req 19845120->19849215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.444967:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19845120->19849215] (req 19845120->19849215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.444976:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396207e38 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19845120->19849215] (req 19845120->19849215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760723.445143:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484183040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52488us (53337us total) trans 4294972163 rc 0/0 00000100:00100000:0.0:1789760723.445157:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4874 00000100:00100000:0.0:1789760723.445170:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712652626335 time 712723564678 state 0x2 ref 0 00000100:00100000:1.0:1789760723.458305:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760723.458542:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484183168 00000100:00100000:0.0:1789760723.458557:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484183168 00000100:00100000:0.0:1789760723.458641:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712652626335 time 712723564678 state 0x0 ref 1 00000100:00100000:0.0:1789760723.458660:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712723825094 time 712723564678 state 0x1 ref 1 00000100:00100000:0.0:1789760723.458704:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712723825094 time 712789634115 state 0x0 ref 1 00000100:00100000:0.0:1789760723.458720:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484183168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760723.458784:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19849216, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:0.0:1789760723.458794:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.458813:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:0.0:1789760723.458818:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760723.458825:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19849216->19853311] (req 19849216->19853311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.459014:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19849216->19853311] (req 19849216->19853311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760723.517641:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972164 is committed 00002000:00010000:2.0:1789760723.517977:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19849216 -> 19853312 00010000:00010000:2.0:1789760723.517997:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19849216->19853311] (req 19849216->19853311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.518013:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19849216->19853311] (req 19849216->19853311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.518024:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19849216->19853311] (req 19849216->19853311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.518032:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760723.518038:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19849216->19853311] (req 19849216->19853311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.518049:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19849216->19853311] (req 19849216->19853311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.518103:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19849216->19853311] (req 19849216->19853311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.518113:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207e3f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19849216->19853311] (req 19849216->19853311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760723.518334:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484183168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59519us (59947us total) trans 4294972164 rc 0/0 00000100:00100000:2.0:1789760723.518347:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4875 00000100:00100000:2.0:1789760723.518360:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712723825094 time 712789634115 state 0x2 ref 0 00000100:00100000:1.0:1789760723.522235:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760723.522644:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484183296 00000100:00100000:3.0:1789760723.522658:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484183296 00000100:00100000:0.0:1789760723.523302:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712723825094 time 712789634115 state 0x0 ref 1 00000100:00100000:0.0:1789760723.523333:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712789894531 time 712789634115 state 0x1 ref 1 00000100:00100000:1.0:1789760723.523474:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712789894531 time 712854402572 state 0x0 ref 1 00000100:00100000:1.0:1789760723.523495:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484183296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760723.523563:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19853312, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:1.0:1789760723.523574:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.523594:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:1.0:1789760723.523599:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760723.523607:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19853312->19857407] (req 19853312->19857407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.523623:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19853312->19857407] (req 19853312->19857407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760723.582997:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972165 is committed 00002000:00010000:1.0:1789760723.583162:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19853312 -> 19857408 00010000:00010000:1.0:1789760723.583178:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19853312->19857407] (req 19853312->19857407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.583190:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19853312->19857407] (req 19853312->19857407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.583200:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19853312->19857407] (req 19853312->19857407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.583207:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760723.583212:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19853312->19857407] (req 19853312->19857407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.583221:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19853312->19857407] (req 19853312->19857407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.583242:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19853312->19857407] (req 19853312->19857407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.583250:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396207e46 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19853312->19857407] (req 19853312->19857407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760723.583338:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484183296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59846us (61115us total) trans 4294972165 rc 0/0 00000100:00100000:1.0:1789760723.583347:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4876 00000100:00100000:1.0:1789760723.583358:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712789894531 time 712854402572 state 0x2 ref 0 00000100:00100000:3.0:1789760723.588104:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760723.588240:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484183424 00000100:00100000:2.0:1789760723.588262:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484183424 00000100:00100000:2.0:1789760723.588348:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712789894531 time 712854402572 state 0x0 ref 1 00000100:00100000:2.0:1789760723.588368:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712854662988 time 712854402572 state 0x1 ref 1 00000100:00100000:2.0:1789760723.588413:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712854662988 time 712919342815 state 0x0 ref 1 00000100:00100000:2.0:1789760723.588431:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484183424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760723.588519:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19857408, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:2.0:1789760723.588532:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.589636:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:2.0:1789760723.589646:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760723.589656:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19857408->19861503] (req 19857408->19861503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760723.589684:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19857408->19861503] (req 19857408->19861503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760723.632056:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972166 is committed 00002000:00010000:3.0:1789760723.632465:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19857408 -> 19861504 00010000:00010000:3.0:1789760723.632481:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19857408->19861503] (req 19857408->19861503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.632496:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19857408->19861503] (req 19857408->19861503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.632507:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19857408->19861503] (req 19857408->19861503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.632515:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760723.632519:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19857408->19861503] (req 19857408->19861503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.632529:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19857408->19861503] (req 19857408->19861503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.632549:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19857408->19861503] (req 19857408->19861503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.632557:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396207e4d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19857408->19861503] (req 19857408->19861503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760723.632750:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484183424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44319us (44683us total) trans 4294972166 rc 0/0 00000100:00100000:3.0:1789760723.632763:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4877 00000100:00100000:3.0:1789760723.632773:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712854662988 time 712919342815 state 0x2 ref 0 00000100:00100000:2.0:1789760723.635574:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760723.635677:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484183552 00000100:00100000:0.0:1789760723.635687:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484183552 00000100:00100000:0.0:1789760723.635748:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712854662988 time 712919342815 state 0x0 ref 1 00000100:00100000:0.0:1789760723.635763:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712919603231 time 712919342815 state 0x1 ref 1 00000100:00100000:0.0:1789760723.635795:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712919603231 time 712966726599 state 0x0 ref 1 00000100:00100000:0.0:1789760723.635809:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484183552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760723.635868:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19861504, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:0.0:1789760723.635876:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.635892:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=712s v=(0 0) 00010000:00010000:0.0:1789760723.635896:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760723.635902:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19861504->19865599] (req 19861504->19865599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.635916:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19861504->19865599] (req 19861504->19865599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760723.667657:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19861504 -> 19865600 00010000:00010000:1.0:1789760723.667676:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19861504->19865599] (req 19861504->19865599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.667690:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19861504->19865599] (req 19861504->19865599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.667699:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19861504->19865599] (req 19861504->19865599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.667706:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760723.667711:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19861504->19865599] (req 19861504->19865599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.667721:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19861504->19865599] (req 19861504->19865599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.667738:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19861504->19865599] (req 19861504->19865599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.667746:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207e54 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19861504->19865599] (req 19861504->19865599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760723.667963:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484183552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32151us (32391us total) trans 4294972167 rc 0/0 00000100:00100000:1.0:1789760723.667975:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4878 00000100:00100000:1.0:1789760723.667985:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712919603231 time 712966726599 state 0x2 ref 0 00000001:00080000:3.0:1789760723.668547:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972167 is committed 00000100:00100000:1.0:1789760723.674189:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760723.674299:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484183680 00000100:00100000:0.0:1789760723.674308:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484183680 00000100:00100000:0.0:1789760723.674379:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712919603231 time 712966726599 state 0x0 ref 1 00000100:00100000:0.0:1789760723.674399:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712966987015 time 712966726599 state 0x1 ref 1 00000100:00100000:0.0:1789760723.674429:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712966987015 time 713005359871 state 0x0 ref 1 00000100:00100000:0.0:1789760723.674443:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484183680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760723.674506:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19865600, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:0.0:1789760723.674515:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.674531:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:0.0:1789760723.674536:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760723.674542:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19865600->19869695] (req 19865600->19869695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.674565:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19865600->19869695] (req 19865600->19869695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760723.717566:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19865600 -> 19869696 00010000:00010000:1.0:1789760723.717593:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19865600->19869695] (req 19865600->19869695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.717609:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19865600->19869695] (req 19865600->19869695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.717618:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19865600->19869695] (req 19865600->19869695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.717627:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760723.717634:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19865600->19869695] (req 19865600->19869695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.717645:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19865600->19869695] (req 19865600->19869695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.717671:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19865600->19869695] (req 19865600->19869695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.717680:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e5b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19865600->19869695] (req 19865600->19869695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760723.717804:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484183680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43357us (43618us total) trans 4294972168 rc 0/0 00000100:00100000:1.0:1789760723.717822:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4879 00000100:00100000:1.0:1789760723.717834:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712966987015 time 713005359871 state 0x2 ref 0 00000001:00080000:3.0:1789760723.717952:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972168 is committed 00000100:00100000:2.0:1789760723.720687:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760723.720776:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484183808 00000100:00100000:1.0:1789760723.720789:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484183808 00000100:00100000:1.0:1789760723.720862:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 712966987015 time 713005359871 state 0x0 ref 1 00000100:00100000:1.0:1789760723.720882:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713005620287 time 713005359871 state 0x1 ref 1 00000100:00100000:1.0:1789760723.720930:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713005620287 time 713051853547 state 0x0 ref 1 00000100:00100000:1.0:1789760723.720945:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484183808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760723.721011:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19869696, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:1.0:1789760723.721022:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.721040:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:1.0:1789760723.721046:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760723.721053:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19869696->19873791] (req 19869696->19873791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.721102:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19869696->19873791] (req 19869696->19873791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760723.755517:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972169 is committed 00002000:00010000:0.0:1789760723.755689:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19869696 -> 19873792 00010000:00010000:0.0:1789760723.755708:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19869696->19873791] (req 19869696->19873791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.755723:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19869696->19873791] (req 19869696->19873791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.755734:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19869696->19873791] (req 19869696->19873791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.755747:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760723.755753:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19869696->19873791] (req 19869696->19873791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.755764:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19869696->19873791] (req 19869696->19873791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.755790:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19869696->19873791] (req 19869696->19873791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.755800:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396207e62 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19869696->19873791] (req 19869696->19873791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760723.755960:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484183808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35011us (35279us total) trans 4294972169 rc 0/0 00000100:00100000:0.0:1789760723.755972:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4880 00000100:00100000:0.0:1789760723.755985:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713005620287 time 713051853547 state 0x2 ref 0 00000100:00100000:2.0:1789760723.761328:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760723.761522:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484183936 00000100:00100000:3.0:1789760723.761542:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484183936 00000100:00100000:3.0:1789760723.761631:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713005620287 time 713051853547 state 0x0 ref 1 00000100:00100000:3.0:1789760723.761649:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713052113963 time 713051853547 state 0x1 ref 1 00000100:00100000:3.0:1789760723.761752:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713052113963 time 713092681777 state 0x0 ref 1 00000100:00100000:3.0:1789760723.761770:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484183936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760723.761833:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19873792, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:3.0:1789760723.761844:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.761863:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:3.0:1789760723.761869:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760723.761877:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19873792->19877887] (req 19873792->19877887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.761894:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19873792->19877887] (req 19873792->19877887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760723.793330:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972170 is committed 00002000:00010000:3.0:1789760723.793485:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19873792 -> 19877888 00010000:00010000:3.0:1789760723.793504:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19873792->19877887] (req 19873792->19877887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.793517:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19873792->19877887] (req 19873792->19877887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.793533:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19873792->19877887] (req 19873792->19877887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.793540:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760723.793546:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19873792->19877887] (req 19873792->19877887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.793555:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19873792->19877887] (req 19873792->19877887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.793580:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19873792->19877887] (req 19873792->19877887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.793589:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e69 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19873792->19877887] (req 19873792->19877887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760723.793720:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484183936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31948us (32400us total) trans 4294972170 rc 0/0 00000100:00100000:3.0:1789760723.793731:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4881 00000100:00100000:3.0:1789760723.793743:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713052113963 time 713092681777 state 0x2 ref 0 00000100:00100000:1.0:1789760723.796929:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760723.797390:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484184064 00000100:00100000:0.0:1789760723.797404:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484184064 00000100:00100000:0.0:1789760723.797680:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713052113963 time 713092681777 state 0x0 ref 1 00000100:00100000:0.0:1789760723.797699:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713092942193 time 713092681777 state 0x1 ref 1 00000100:00100000:0.0:1789760723.797736:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713092942193 time 713128666551 state 0x0 ref 1 00000100:00100000:0.0:1789760723.797755:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484184064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760723.797826:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19877888, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:0.0:1789760723.797837:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.797858:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:0.0:1789760723.797863:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760723.797870:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19877888->19881983] (req 19877888->19881983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.797887:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19877888->19881983] (req 19877888->19881983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760723.836905:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19877888 -> 19881984 00010000:00010000:0.0:1789760723.836934:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19877888->19881983] (req 19877888->19881983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.836949:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19877888->19881983] (req 19877888->19881983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.836959:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19877888->19881983] (req 19877888->19881983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.836966:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760723.836971:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19877888->19881983] (req 19877888->19881983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.836981:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19877888->19881983] (req 19877888->19881983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.837005:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19877888->19881983] (req 19877888->19881983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760723.837012:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e70 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19877888->19881983] (req 19877888->19881983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760723.837302:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484184064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39545us (40375us total) trans 4294972171 rc 0/0 00000100:00100000:0.0:1789760723.837316:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4882 00000100:00100000:0.0:1789760723.837330:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713092942193 time 713128666551 state 0x2 ref 0 00000001:00080000:2.0:1789760723.837611:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972171 is committed 00000100:00100000:2.0:1789760723.843102:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760723.843321:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484184192 00000100:00100000:1.0:1789760723.843335:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484184192 00000100:00100000:1.0:1789760723.843407:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713092942193 time 713128666551 state 0x0 ref 1 00000100:00100000:1.0:1789760723.843426:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713128926967 time 713128666551 state 0x1 ref 1 00000100:00100000:1.0:1789760723.843520:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713128926967 time 713174450309 state 0x0 ref 1 00000100:00100000:1.0:1789760723.843535:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484184192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760723.843599:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19881984, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:1.0:1789760723.843610:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.843631:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:1.0:1789760723.843636:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760723.843643:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19881984->19886079] (req 19881984->19886079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.843659:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19881984->19886079] (req 19881984->19886079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.852498:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:1.0:1789760723.883935:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19881984 -> 19886080 00000001:00080000:0.0:1789760723.883943:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972172 is committed 00010000:00010000:1.0:1789760723.883960:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19881984->19886079] (req 19881984->19886079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.883975:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19881984->19886079] (req 19881984->19886079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.883996:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19881984->19886079] (req 19881984->19886079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.884004:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760723.884009:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19881984->19886079] (req 19881984->19886079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.884019:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19881984->19886079] (req 19881984->19886079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.884044:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19881984->19886079] (req 19881984->19886079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.884052:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207e77 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19881984->19886079] (req 19881984->19886079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760723.884339:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484184192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40797us (41278us total) trans 4294972172 rc 0/0 00000100:00100000:1.0:1789760723.884353:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4883 00000100:00100000:1.0:1789760723.884367:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713128926967 time 713174450309 state 0x2 ref 0 00000100:00100000:3.0:1789760723.888666:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760723.888875:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484184320 00000100:00100000:1.0:1789760723.888889:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484184320 00000100:00100000:1.0:1789760723.888963:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713128926967 time 713174450309 state 0x0 ref 1 00000100:00100000:1.0:1789760723.888981:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713174710725 time 713174450309 state 0x1 ref 1 00000100:00100000:1.0:1789760723.889020:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713174710725 time 713219946674 state 0x0 ref 1 00000100:00100000:1.0:1789760723.889036:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484184320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760723.889125:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19886080, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:1.0:1789760723.889137:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.889157:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:1.0:1789760723.889288:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760723.889296:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19886080->19890175] (req 19886080->19890175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.889315:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19886080->19890175] (req 19886080->19890175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760723.931214:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972173 is committed 00002000:00010000:1.0:1789760723.931410:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19886080 -> 19890176 00010000:00010000:1.0:1789760723.931432:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19886080->19890175] (req 19886080->19890175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.931458:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19886080->19890175] (req 19886080->19890175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.931469:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19886080->19890175] (req 19886080->19890175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.931476:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760723.931482:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19886080->19890175] (req 19886080->19890175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.931492:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19886080->19890175] (req 19886080->19890175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.931518:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19886080->19890175] (req 19886080->19890175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760723.931527:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e7e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19886080->19890175] (req 19886080->19890175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760723.931660:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484184320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42621us (43003us total) trans 4294972173 rc 0/0 00000100:00100000:1.0:1789760723.931856:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4884 00000100:00100000:1.0:1789760723.931869:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713174710725 time 713219946674 state 0x2 ref 0 00000100:00100000:2.0:1789760723.937230:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760723.937458:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484184448 00000100:00100000:3.0:1789760723.937471:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484184448 00000100:00100000:3.0:1789760723.937958:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713174710725 time 713219946674 state 0x0 ref 1 00000100:00100000:3.0:1789760723.937978:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713220207090 time 713219946674 state 0x1 ref 1 00000100:00100000:3.0:1789760723.938117:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713220207090 time 713269045968 state 0x0 ref 1 00000100:00100000:3.0:1789760723.938136:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484184448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760723.938212:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19890176, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:3.0:1789760723.938224:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.938253:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:3.0:1789760723.938258:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760723.938265:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19890176->19894271] (req 19890176->19894271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760723.938282:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19890176->19894271] (req 19890176->19894271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760724.003057:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972174 is committed 00002000:00010000:2.0:1789760724.003147:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19890176 -> 19894272 00010000:00010000:2.0:1789760724.003187:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19890176->19894271] (req 19890176->19894271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.003205:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19890176->19894271] (req 19890176->19894271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.003217:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19890176->19894271] (req 19890176->19894271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.003226:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.003232:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19890176->19894271] (req 19890176->19894271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.003243:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19890176->19894271] (req 19890176->19894271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.003285:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19890176->19894271] (req 19890176->19894271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.003294:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207e85 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19890176->19894271] (req 19890176->19894271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.003421:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484184448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65284us (66200us total) trans 4294972174 rc 0/0 00000100:00100000:2.0:1789760724.003434:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4885 00000100:00100000:2.0:1789760724.003454:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713220207090 time 713269045968 state 0x2 ref 0 00000100:00100000:2.0:1789760724.009183:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760724.009289:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484184576 00000100:00100000:0.0:1789760724.009302:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484184576 00000100:00100000:0.0:1789760724.009391:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713220207090 time 713269045968 state 0x0 ref 1 00000100:00100000:0.0:1789760724.009407:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713269306384 time 713269045968 state 0x1 ref 1 00000100:00100000:0.0:1789760724.009443:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713269306384 time 713340373452 state 0x0 ref 1 00000100:00100000:0.0:1789760724.009479:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484184576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760724.009545:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19894272, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:0.0:1789760724.009556:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.009575:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:0.0:1789760724.009580:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760724.009588:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19894272->19898367] (req 19894272->19898367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.009605:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19894272->19898367] (req 19894272->19898367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760724.010173:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760724.041481:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972175 is committed 00002000:00010000:3.0:1789760724.041544:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19894272 -> 19898368 00010000:00010000:3.0:1789760724.041564:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19894272->19898367] (req 19894272->19898367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.041579:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19894272->19898367] (req 19894272->19898367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.041597:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19894272->19898367] (req 19894272->19898367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.041606:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760724.041613:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19894272->19898367] (req 19894272->19898367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.041623:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19894272->19898367] (req 19894272->19898367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.041651:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19894272->19898367] (req 19894272->19898367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.041661:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207e8c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19894272->19898367] (req 19894272->19898367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760724.041783:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484184576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32309us (32605us total) trans 4294972175 rc 0/0 00000100:00100000:3.0:1789760724.041795:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4886 00000100:00100000:3.0:1789760724.041807:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713269306384 time 713340373452 state 0x2 ref 0 00000100:00100000:2.0:1789760724.046931:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760724.047229:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484184704 00000100:00100000:1.0:1789760724.047242:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484184704 00000100:00100000:1.0:1789760724.047318:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713269306384 time 713340373452 state 0x0 ref 1 00000100:00100000:1.0:1789760724.047336:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713340633868 time 713340373452 state 0x1 ref 1 00000100:00100000:1.0:1789760724.047428:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713340633868 time 713378358171 state 0x0 ref 1 00000100:00100000:1.0:1789760724.047461:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484184704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760724.047527:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19898368, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:1.0:1789760724.047537:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.047567:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:1.0:1789760724.047572:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760724.047580:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19898368->19902463] (req 19898368->19902463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.047596:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19898368->19902463] (req 19898368->19902463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760724.074329:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972176 is committed 00002000:00010000:1.0:1789760724.074579:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19898368 -> 19902464 00010000:00010000:1.0:1789760724.074595:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19898368->19902463] (req 19898368->19902463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.074609:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19898368->19902463] (req 19898368->19902463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.074618:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19898368->19902463] (req 19898368->19902463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.074625:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760724.074630:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19898368->19902463] (req 19898368->19902463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.074639:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19898368->19902463] (req 19898368->19902463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.074665:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19898368->19902463] (req 19898368->19902463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.074673:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207e93 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19898368->19902463] (req 19898368->19902463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760724.074768:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484184704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27322us (27848us total) trans 4294972176 rc 0/0 00000100:00100000:1.0:1789760724.074778:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4887 00000100:00100000:1.0:1789760724.074787:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713340633868 time 713378358171 state 0x2 ref 0 00000100:00100000:2.0:1789760724.079858:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760724.079982:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484184832 00000100:00100000:1.0:1789760724.079991:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484184832 00000100:00100000:1.0:1789760724.080038:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713340633868 time 713378358171 state 0x0 ref 1 00000100:00100000:1.0:1789760724.080061:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713378618587 time 713378358171 state 0x1 ref 1 00000100:00100000:3.0:1789760724.080226:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713378618587 time 713411155464 state 0x0 ref 1 00000100:00100000:3.0:1789760724.080240:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484184832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760724.080286:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19902464, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:3.0:1789760724.080294:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.080309:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:3.0:1789760724.080313:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760724.080319:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19902464->19906559] (req 19902464->19906559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.080332:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19902464->19906559] (req 19902464->19906559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760724.138146:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972177 is committed 00002000:00010000:2.0:1789760724.138265:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19902464 -> 19906560 00010000:00010000:2.0:1789760724.138289:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19902464->19906559] (req 19902464->19906559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.138304:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19902464->19906559] (req 19902464->19906559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.138315:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19902464->19906559] (req 19902464->19906559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.138329:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.138336:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19902464->19906559] (req 19902464->19906559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.138346:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19902464->19906559] (req 19902464->19906559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.138369:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19902464->19906559] (req 19902464->19906559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.138377:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207e9a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19902464->19906559] (req 19902464->19906559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.138510:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484184832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58266us (58653us total) trans 4294972177 rc 0/0 00000100:00100000:2.0:1789760724.138524:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4888 00000100:00100000:2.0:1789760724.138768:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713378618587 time 713411155464 state 0x2 ref 0 00000100:00100000:2.0:1789760724.141774:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760724.142007:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484184960 00000100:00100000:0.0:1789760724.142021:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484184960 00000100:00100000:0.0:1789760724.142136:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713378618587 time 713411155464 state 0x0 ref 1 00000100:00100000:0.0:1789760724.142154:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713411415880 time 713411155464 state 0x1 ref 1 00000100:00100000:0.0:1789760724.142200:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713411415880 time 713473130470 state 0x0 ref 1 00000100:00100000:0.0:1789760724.142216:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484184960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760724.142286:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19906560, mtime: 0x6aad94d3, atime: 0x0, ctime: 0x6aad94d3, blocks: 0x9807 00010000:00010000:0.0:1789760724.142297:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.142318:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:0.0:1789760724.142324:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760724.142331:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19906560->19910655] (req 19906560->19910655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.142348:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19906560->19910655] (req 19906560->19910655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760724.170654:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972178 is committed 00002000:00010000:0.0:1789760724.170667:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19906560 -> 19910656 00002000:00010000:0.0:1789760724.170672:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760723 -> 1789760724 00002000:00010000:0.0:1789760724.170675:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760723 -> 1789760724 00010000:00010000:0.0:1789760724.170692:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19906560->19910655] (req 19906560->19910655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.170706:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19906560->19910655] (req 19906560->19910655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.170716:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19906560->19910655] (req 19906560->19910655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.170725:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760724.170730:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19906560->19910655] (req 19906560->19910655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.170741:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19906560->19910655] (req 19906560->19910655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.170764:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19906560->19910655] (req 19906560->19910655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.170774:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ea1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19906560->19910655] (req 19906560->19910655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760724.170904:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484184960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28685us (29137us total) trans 4294972178 rc 0/0 00000100:00100000:0.0:1789760724.170916:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4889 00000100:00100000:0.0:1789760724.170929:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713411415880 time 713473130470 state 0x2 ref 0 00000100:00100000:3.0:1789760724.173616:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760724.174618:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484185088 00000100:00100000:0.0:1789760724.174635:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484185088 00000100:00100000:0.0:1789760724.174733:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713411415880 time 713473130470 state 0x0 ref 1 00000100:00100000:0.0:1789760724.174753:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713473390886 time 713473130470 state 0x1 ref 1 00000100:00100000:0.0:1789760724.174857:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713473390886 time 713505786769 state 0x0 ref 1 00000100:00100000:0.0:1789760724.174876:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484185088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760724.174954:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19910656, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0x9807 00010000:00010000:0.0:1789760724.174965:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.174987:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:0.0:1789760724.174992:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760724.175000:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19910656->19914751] (req 19910656->19914751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.175023:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19910656->19914751] (req 19910656->19914751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760724.215267:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972179 is committed 00002000:00010000:0.0:1789760724.215417:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19910656 -> 19914752 00010000:00010000:0.0:1789760724.215438:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19910656->19914751] (req 19910656->19914751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.215462:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19910656->19914751] (req 19910656->19914751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.215471:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19910656->19914751] (req 19910656->19914751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.215479:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760724.215484:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19910656->19914751] (req 19910656->19914751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.215495:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19910656->19914751] (req 19910656->19914751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.215519:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19910656->19914751] (req 19910656->19914751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.215529:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396207ea8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19910656->19914751] (req 19910656->19914751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760724.215823:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484185088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40945us (42212us total) trans 4294972179 rc 0/0 00000100:00100000:0.0:1789760724.215836:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4890 00000100:00100000:0.0:1789760724.215923:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713473390886 time 713505786769 state 0x2 ref 0 00000100:00100000:2.0:1789760724.220538:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760724.220864:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484185216 00000100:00100000:1.0:1789760724.220877:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484185216 00000100:00100000:1.0:1789760724.220952:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713473390886 time 713505786769 state 0x0 ref 1 00000100:00100000:1.0:1789760724.220973:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713506047185 time 713505786769 state 0x1 ref 1 00000100:00100000:1.0:1789760724.221006:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713506047185 time 713551937400 state 0x0 ref 1 00000100:00100000:1.0:1789760724.221020:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484185216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760724.221115:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19914752, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0x9807 00010000:00010000:1.0:1789760724.221127:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.221147:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:1.0:1789760724.221152:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760724.221159:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19914752->19918847] (req 19914752->19918847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.221186:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19914752->19918847] (req 19914752->19918847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760724.271177:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972180 is committed 00002000:00010000:2.0:1789760724.272979:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19914752 -> 19918848 00010000:00010000:2.0:1789760724.273003:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19914752->19918847] (req 19914752->19918847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.273024:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19914752->19918847] (req 19914752->19918847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.273035:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19914752->19918847] (req 19914752->19918847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.273043:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.273048:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19914752->19918847] (req 19914752->19918847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.273058:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19914752->19918847] (req 19914752->19918847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.273138:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19914752->19918847] (req 19914752->19918847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.273148:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396207eaf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19914752->19918847] (req 19914752->19918847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.273675:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484185216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52650us (53145us total) trans 4294972180 rc 0/0 00000100:00100000:2.0:1789760724.273688:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4891 00000100:00100000:2.0:1789760724.273702:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713506047185 time 713551937400 state 0x2 ref 0 00000100:00100000:2.0:1789760724.282542:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760724.282651:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484185344 00000100:00100000:3.0:1789760724.282664:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484185344 00000100:00100000:3.0:1789760724.282732:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713506047185 time 713551937400 state 0x0 ref 1 00000100:00100000:3.0:1789760724.282750:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713552197816 time 713551937400 state 0x1 ref 1 00000100:00100000:3.0:1789760724.282776:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713552197816 time 713613706863 state 0x0 ref 1 00000100:00100000:3.0:1789760724.282789:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484185344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760724.283360:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19918848, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0x9807 00010000:00010000:3.0:1789760724.283378:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.283400:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:3.0:1789760724.283406:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760724.283413:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19918848->19922943] (req 19918848->19922943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.283432:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19918848->19922943] (req 19918848->19922943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760724.327721:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19918848 -> 19922944 00010000:00010000:2.0:1789760724.327749:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19918848->19922943] (req 19918848->19922943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.327764:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19918848->19922943] (req 19918848->19922943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760724.327772:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972181 is committed 00010000:00010000:2.0:1789760724.327773:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19918848->19922943] (req 19918848->19922943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.327780:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.327785:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19918848->19922943] (req 19918848->19922943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.327795:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19918848->19922943] (req 19918848->19922943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.327821:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19918848->19922943] (req 19918848->19922943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.327829:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207eb6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19918848->19922943] (req 19918848->19922943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.327994:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484185344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45200us (45462us total) trans 4294972181 rc 0/0 00000100:00100000:2.0:1789760724.328004:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4892 00000100:00100000:2.0:1789760724.328015:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713552197816 time 713613706863 state 0x2 ref 0 00000100:00100000:3.0:1789760724.331300:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760724.332125:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484185472 00000100:00100000:0.0:1789760724.332140:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484185472 00000100:00100000:0.0:1789760724.332234:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713552197816 time 713613706863 state 0x0 ref 1 00000100:00100000:0.0:1789760724.332312:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713613967279 time 713613706863 state 0x1 ref 1 00000100:00100000:0.0:1789760724.332349:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713613967279 time 713663279202 state 0x0 ref 1 00000100:00100000:0.0:1789760724.332365:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484185472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760724.332437:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19922944, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0x9807 00010000:00010000:0.0:1789760724.332456:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.332475:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:0.0:1789760724.332479:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760724.332487:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19922944->19927039] (req 19922944->19927039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.332504:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19922944->19927039] (req 19922944->19927039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760724.388552:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19922944 -> 19927040 00002000:00010000:1.0:1789760724.388567:0:14643:0:(ofd_lvb.c:331:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb blocks from disk: 38919 -> 40967 00010000:00010000:1.0:1789760724.388585:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19922944->19927039] (req 19922944->19927039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.388600:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19922944->19927039] (req 19922944->19927039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.388610:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19922944->19927039] (req 19922944->19927039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.388618:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760724.388623:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19922944->19927039] (req 19922944->19927039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.388635:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19922944->19927039] (req 19922944->19927039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.388661:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19922944->19927039] (req 19922944->19927039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.388670:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207ebd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19922944->19927039] (req 19922944->19927039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760724.388797:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484185472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56429us (57504us total) trans 4294972182 rc 0/0 00000100:00100000:1.0:1789760724.388810:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4893 00000100:00100000:1.0:1789760724.388822:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713613967279 time 713663279202 state 0x2 ref 0 00000001:00080000:3.0:1789760724.389104:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972182 is committed 00000100:00100000:0.0:1789760724.391778:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760724.391980:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484185600 00000100:00100000:2.0:1789760724.391994:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484185600 00000100:00100000:2.0:1789760724.392376:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713613967279 time 713663279202 state 0x0 ref 1 00000100:00100000:2.0:1789760724.392398:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713663539618 time 713663279202 state 0x1 ref 1 00000100:00100000:2.0:1789760724.392484:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713663539618 time 713723413664 state 0x0 ref 1 00000100:00100000:2.0:1789760724.392499:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484185600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760724.392567:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19927040, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:2.0:1789760724.392578:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.392598:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:2.0:1789760724.392604:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760724.392611:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19927040->19931135] (req 19927040->19931135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.392626:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19927040->19931135] (req 19927040->19931135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760724.422572:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972183 is committed 00002000:00010000:2.0:1789760724.422707:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19927040 -> 19931136 00010000:00010000:2.0:1789760724.422728:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19927040->19931135] (req 19927040->19931135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.422742:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19927040->19931135] (req 19927040->19931135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.422754:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19927040->19931135] (req 19927040->19931135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.422762:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.422768:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19927040->19931135] (req 19927040->19931135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.422778:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19927040->19931135] (req 19927040->19931135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.422804:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19927040->19931135] (req 19927040->19931135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.422812:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ec4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19927040->19931135] (req 19927040->19931135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.422970:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484185600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30468us (31198us total) trans 4294972183 rc 0/0 00000100:00100000:2.0:1789760724.422982:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4894 00000100:00100000:2.0:1789760724.422995:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713663539618 time 713723413664 state 0x2 ref 0 00000100:00100000:2.0:1789760724.428647:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760724.428817:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484185728 00000100:00100000:1.0:1789760724.428831:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484185728 00000100:00100000:1.0:1789760724.428901:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713663539618 time 713723413664 state 0x0 ref 1 00000100:00100000:1.0:1789760724.428919:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713723674080 time 713723413664 state 0x1 ref 1 00000100:00100000:1.0:1789760724.429000:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713723674080 time 713759930737 state 0x0 ref 1 00000100:00100000:1.0:1789760724.429015:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484185728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760724.429108:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19931136, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:1.0:1789760724.429120:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.429139:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:1.0:1789760724.429144:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760724.429151:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19931136->19935231] (req 19931136->19935231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.429167:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19931136->19935231] (req 19931136->19935231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760724.475109:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972184 is committed 00002000:00010000:2.0:1789760724.475269:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19931136 -> 19935232 00010000:00010000:2.0:1789760724.475290:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19931136->19935231] (req 19931136->19935231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.475312:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19931136->19935231] (req 19931136->19935231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.475322:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19931136->19935231] (req 19931136->19935231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.475330:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.475336:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19931136->19935231] (req 19931136->19935231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.475345:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19931136->19935231] (req 19931136->19935231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.475371:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19931136->19935231] (req 19931136->19935231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.475379:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ecb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19931136->19935231] (req 19931136->19935231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.475510:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484185728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46490us (46871us total) trans 4294972184 rc 0/0 00000100:00100000:2.0:1789760724.475522:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4895 00000100:00100000:2.0:1789760724.475543:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713723674080 time 713759930737 state 0x2 ref 0 00000100:00100000:3.0:1789760724.480979:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760724.485781:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484185856 00000100:00100000:2.0:1789760724.485805:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484185856 00000100:00100000:2.0:1789760724.485891:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713723674080 time 713759930737 state 0x0 ref 1 00000100:00100000:2.0:1789760724.485911:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713760191153 time 713759930737 state 0x1 ref 1 00000100:00100000:2.0:1789760724.485950:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713760191153 time 713816881244 state 0x0 ref 1 00000100:00100000:2.0:1789760724.485967:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484185856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760724.486033:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19935232, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:2.0:1789760724.486043:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.486085:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:2.0:1789760724.486091:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760724.486100:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19935232->19939327] (req 19935232->19939327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.486117:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19935232->19939327] (req 19935232->19939327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760724.517600:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972185 is committed 00002000:00010000:2.0:1789760724.517848:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19935232 -> 19939328 00010000:00010000:2.0:1789760724.517868:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19935232->19939327] (req 19935232->19939327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.517883:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19935232->19939327] (req 19935232->19939327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.517899:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19935232->19939327] (req 19935232->19939327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.517907:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.517913:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19935232->19939327] (req 19935232->19939327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.517923:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19935232->19939327] (req 19935232->19939327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.517950:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19935232->19939327] (req 19935232->19939327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.517958:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ed2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19935232->19939327] (req 19935232->19939327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.518173:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484185856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32203us (37203us total) trans 4294972185 rc 0/0 00000100:00100000:2.0:1789760724.518186:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4896 00000100:00100000:2.0:1789760724.518199:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713760191153 time 713816881244 state 0x2 ref 0 00000100:00100000:0.0:1789760724.525025:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760724.525563:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484185984 00000100:00100000:1.0:1789760724.525578:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484185984 00000100:00100000:1.0:1789760724.525657:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713760191153 time 713816881244 state 0x0 ref 1 00000100:00100000:1.0:1789760724.525676:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713817141660 time 713816881244 state 0x1 ref 1 00000100:00100000:1.0:1789760724.525750:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713817141660 time 713856679690 state 0x0 ref 1 00000100:00100000:1.0:1789760724.525766:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484185984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760724.525833:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19939328, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:1.0:1789760724.525843:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.525863:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:1.0:1789760724.525868:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760724.525876:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19939328->19943423] (req 19939328->19943423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.525894:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19939328->19943423] (req 19939328->19943423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760724.568092:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972186 is committed 00002000:00010000:0.0:1789760724.569098:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19939328 -> 19943424 00010000:00010000:0.0:1789760724.569117:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19939328->19943423] (req 19939328->19943423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.569132:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19939328->19943423] (req 19939328->19943423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.569142:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19939328->19943423] (req 19939328->19943423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.569149:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760724.569293:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19939328->19943423] (req 19939328->19943423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.569306:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19939328->19943423] (req 19939328->19943423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.569331:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19939328->19943423] (req 19939328->19943423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.569340:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207ed9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19939328->19943423] (req 19939328->19943423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760724.569527:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484185984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43758us (44510us total) trans 4294972186 rc 0/0 00000100:00100000:0.0:1789760724.569540:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4897 00000100:00100000:0.0:1789760724.569552:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713817141660 time 713856679690 state 0x2 ref 0 00000100:00100000:3.0:1789760724.573405:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760724.574537:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484186112 00000100:00100000:2.0:1789760724.574554:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484186112 00000100:00100000:2.0:1789760724.574626:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713817141660 time 713856679690 state 0x0 ref 1 00000100:00100000:2.0:1789760724.574645:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713856940106 time 713856679690 state 0x1 ref 1 00000100:00100000:2.0:1789760724.574711:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713856940106 time 713905641831 state 0x0 ref 1 00000100:00100000:2.0:1789760724.574726:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484186112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760724.574791:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19943424, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:2.0:1789760724.574800:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.575793:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=713s v=(0 0) 00010000:00010000:2.0:1789760724.575804:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760724.575816:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19943424->19947519] (req 19943424->19947519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.575910:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19943424->19947519] (req 19943424->19947519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760724.669308:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972187 is committed 00002000:00010000:2.0:1789760724.669817:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19943424 -> 19947520 00010000:00010000:2.0:1789760724.669837:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19943424->19947519] (req 19943424->19947519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.669853:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19943424->19947519] (req 19943424->19947519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.669863:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19943424->19947519] (req 19943424->19947519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.669871:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.669877:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19943424->19947519] (req 19943424->19947519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.669888:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19943424->19947519] (req 19943424->19947519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.669913:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19943424->19947519] (req 19943424->19947519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.669920:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19943424->19947519] (req 19943424->19947519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.670047:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484186112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 95317us (96648us total) trans 4294972187 rc 0/0 00000100:00100000:2.0:1789760724.670060:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4898 00000100:00100000:2.0:1789760724.670111:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713856940106 time 713905641831 state 0x2 ref 0 00000100:00100000:3.0:1789760724.680225:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760724.680502:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484186240 00000100:00100000:2.0:1789760724.680517:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484186240 00000100:00100000:2.0:1789760724.680612:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713856940106 time 713905641831 state 0x0 ref 1 00000100:00100000:2.0:1789760724.680630:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713905902247 time 713905641831 state 0x1 ref 1 00000100:00100000:2.0:1789760724.680712:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713905902247 time 714011641582 state 0x0 ref 1 00000100:00100000:2.0:1789760724.680741:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484186240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760724.680811:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19947520, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:2.0:1789760724.680823:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.680843:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:2.0:1789760724.680848:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760724.680856:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19947520->19951615] (req 19947520->19951615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.680875:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19947520->19951615] (req 19947520->19951615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760724.733275:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19947520 -> 19951616 00010000:00010000:2.0:1789760724.733302:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19947520->19951615] (req 19947520->19951615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.733318:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19947520->19951615] (req 19947520->19951615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.733327:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19947520->19951615] (req 19947520->19951615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.733336:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.733342:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19947520->19951615] (req 19947520->19951615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.733353:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19947520->19951615] (req 19947520->19951615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.733385:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19947520->19951615] (req 19947520->19951615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.733394:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207ee7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19947520->19951615] (req 19947520->19951615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.733537:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484186240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52794us (53330us total) trans 4294972188 rc 0/0 00000100:00100000:2.0:1789760724.733549:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4899 00000100:00100000:2.0:1789760724.733562:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713905902247 time 714011641582 state 0x2 ref 0 00000001:00080000:1.0:1789760724.734122:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972188 is committed 00000100:00100000:0.0:1789760724.741208:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760724.741328:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484186368 00000100:00100000:2.0:1789760724.741342:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484186368 00000100:00100000:2.0:1789760724.741416:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 713905902247 time 714011641582 state 0x0 ref 1 00000100:00100000:2.0:1789760724.741436:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714011901998 time 714011641582 state 0x1 ref 1 00000100:00100000:0.0:1789760724.741778:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714011901998 time 714072706360 state 0x0 ref 1 00000100:00100000:0.0:1789760724.741804:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484186368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760724.741871:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19951616, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:0.0:1789760724.741882:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.741901:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:0.0:1789760724.741906:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760724.741914:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19951616->19955711] (req 19951616->19955711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.741931:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19951616->19955711] (req 19951616->19955711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760724.783252:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972189 is committed 00002000:00010000:0.0:1789760724.783393:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19951616 -> 19955712 00010000:00010000:0.0:1789760724.783413:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19951616->19955711] (req 19951616->19955711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.783432:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19951616->19955711] (req 19951616->19955711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.783452:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19951616->19955711] (req 19951616->19955711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.783461:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760724.783466:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19951616->19955711] (req 19951616->19955711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.783477:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19951616->19955711] (req 19951616->19955711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.783503:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19951616->19955711] (req 19951616->19955711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.783510:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207eee lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19951616->19955711] (req 19951616->19955711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760724.783746:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484186368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41948us (42546us total) trans 4294972189 rc 0/0 00000100:00100000:0.0:1789760724.783759:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4900 00000100:00100000:0.0:1789760724.783771:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714011901998 time 714072706360 state 0x2 ref 0 00000100:00100000:0.0:1789760724.786582:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760724.788943:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484186496 00000100:00100000:1.0:1789760724.788963:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484186496 00000100:00100000:1.0:1789760724.789049:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714011901998 time 714072706360 state 0x0 ref 1 00000100:00100000:1.0:1789760724.789090:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714072966776 time 714072706360 state 0x1 ref 1 00000100:00100000:1.0:1789760724.789128:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714072966776 time 714120058649 state 0x0 ref 1 00000100:00100000:1.0:1789760724.789144:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484186496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760724.789217:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19955712, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:1.0:1789760724.789228:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.789247:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:1.0:1789760724.789251:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760724.789259:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19955712->19959807] (req 19955712->19959807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.789275:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19955712->19959807] (req 19955712->19959807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760724.843862:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972190 is committed 00002000:00010000:1.0:1789760724.844168:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19955712 -> 19959808 00010000:00010000:1.0:1789760724.844189:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19955712->19959807] (req 19955712->19959807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.844205:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19955712->19959807] (req 19955712->19959807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.844216:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19955712->19959807] (req 19955712->19959807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.844233:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760724.844239:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19955712->19959807] (req 19955712->19959807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.844249:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19955712->19959807] (req 19955712->19959807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.844274:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19955712->19959807] (req 19955712->19959807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760724.844283:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207ef5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19955712->19959807] (req 19955712->19959807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760724.844400:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484186496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55253us (57828us total) trans 4294972190 rc 0/0 00000100:00100000:1.0:1789760724.844415:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4901 00000100:00100000:1.0:1789760724.844427:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714072966776 time 714120058649 state 0x2 ref 0 00000100:00100000:0.0:1789760724.847115:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760724.847394:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484186624 00000100:00100000:3.0:1789760724.847407:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484186624 00000100:00100000:3.0:1789760724.847502:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714072966776 time 714120058649 state 0x0 ref 1 00000100:00100000:3.0:1789760724.847522:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714120319065 time 714120058649 state 0x1 ref 1 00000100:00100000:3.0:1789760724.847593:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714120319065 time 714178523187 state 0x0 ref 1 00000100:00100000:3.0:1789760724.847606:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484186624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760724.847672:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19959808, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:3.0:1789760724.847682:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.847703:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760724.847708:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760724.847716:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19959808->19963903] (req 19959808->19963903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.847734:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19959808->19963903] (req 19959808->19963903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.876581:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760724.881370:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972191 is committed 00002000:00010000:2.0:1789760724.881933:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19959808 -> 19963904 00010000:00010000:2.0:1789760724.881956:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19959808->19963903] (req 19959808->19963903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.881972:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19959808->19963903] (req 19959808->19963903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.881983:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19959808->19963903] (req 19959808->19963903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.881998:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.882004:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19959808->19963903] (req 19959808->19963903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.882014:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19959808->19963903] (req 19959808->19963903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.882040:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19959808->19963903] (req 19959808->19963903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.882048:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207efc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19959808->19963903] (req 19959808->19963903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760724.882523:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484186624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34911us (35451us total) trans 4294972191 rc 0/0 00000100:00100000:0.0:1789760724.882535:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4902 00000100:00100000:0.0:1789760724.882559:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714120319065 time 714178523187 state 0x2 ref 0 00000100:00100000:0.0:1789760724.887846:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760724.888179:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484186752 00000100:00100000:3.0:1789760724.888195:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484186752 00000100:00100000:3.0:1789760724.888272:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714120319065 time 714178523187 state 0x0 ref 1 00000100:00100000:3.0:1789760724.888293:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714178783603 time 714178523187 state 0x1 ref 1 00000100:00100000:3.0:1789760724.888331:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714178783603 time 714219261644 state 0x0 ref 1 00000100:00100000:3.0:1789760724.888346:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484186752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760724.888411:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19963904, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:3.0:1789760724.888423:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.888459:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760724.888465:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760724.888473:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19963904->19967999] (req 19963904->19967999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.888496:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19963904->19967999] (req 19963904->19967999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760724.929155:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19963904 -> 19968000 00010000:00010000:2.0:1789760724.929184:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19963904->19967999] (req 19963904->19967999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.929199:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19963904->19967999] (req 19963904->19967999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.929209:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19963904->19967999] (req 19963904->19967999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.929218:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.929224:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19963904->19967999] (req 19963904->19967999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.929235:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19963904->19967999] (req 19963904->19967999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760724.929238:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972192 is committed 00010000:00010000:2.0:1789760724.929261:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19963904->19967999] (req 19963904->19967999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.929269:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f03 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19963904->19967999] (req 19963904->19967999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.929397:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484186752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41046us (41558us total) trans 4294972192 rc 0/0 00000100:00100000:2.0:1789760724.929409:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4903 00000100:00100000:2.0:1789760724.929421:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714178783603 time 714219261644 state 0x2 ref 0 00000100:00100000:3.0:1789760724.933953:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760724.934193:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484186880 00000100:00100000:0.0:1789760724.934207:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484186880 00000100:00100000:0.0:1789760724.934353:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714178783603 time 714219261644 state 0x0 ref 1 00000100:00100000:0.0:1789760724.934371:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714219522060 time 714219261644 state 0x1 ref 1 00000100:00100000:0.0:1789760724.934409:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714219522060 time 714265338826 state 0x0 ref 1 00000100:00100000:0.0:1789760724.934424:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484186880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760724.934500:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19968000, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:0.0:1789760724.934510:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.934667:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:0.0:1789760724.934673:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760724.934682:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19968000->19972095] (req 19968000->19972095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760724.934706:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19968000->19972095] (req 19968000->19972095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760724.967299:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19968000 -> 19972096 00010000:00010000:2.0:1789760724.967326:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19968000->19972095] (req 19968000->19972095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.967340:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19968000->19972095] (req 19968000->19972095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.967350:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19968000->19972095] (req 19968000->19972095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.967357:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760724.967364:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19968000->19972095] (req 19968000->19972095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.967373:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19968000->19972095] (req 19968000->19972095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.967397:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19968000->19972095] (req 19968000->19972095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760724.967404:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396207f0a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19968000->19972095] (req 19968000->19972095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760724.967530:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484186880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33103us (33585us total) trans 4294972193 rc 0/0 00000100:00100000:2.0:1789760724.967543:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4904 00000100:00100000:2.0:1789760724.967554:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714219522060 time 714265338826 state 0x2 ref 0 00000001:00080000:3.0:1789760724.967631:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972193 is committed 00000100:00100000:0.0:1789760724.970192:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760724.970429:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484187008 00000100:00100000:1.0:1789760724.970452:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484187008 00000100:00100000:2.0:1789760724.970672:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714219522060 time 714265338826 state 0x0 ref 1 00000100:00100000:2.0:1789760724.970693:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714265599242 time 714265338826 state 0x1 ref 1 00000100:00100000:3.0:1789760724.970943:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714265599242 time 714301872208 state 0x0 ref 1 00000100:00100000:3.0:1789760724.970958:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484187008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760724.971013:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19972096, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:3.0:1789760724.971021:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.971037:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760724.971041:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760724.971046:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19972096->19976191] (req 19972096->19976191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760724.971059:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19972096->19976191] (req 19972096->19976191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760725.010204:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972194 is committed 00002000:00010000:3.0:1789760725.010404:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19972096 -> 19976192 00010000:00010000:3.0:1789760725.010485:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19972096->19976191] (req 19972096->19976191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.010501:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19972096->19976191] (req 19972096->19976191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.010511:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19972096->19976191] (req 19972096->19976191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.010520:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760725.010534:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19972096->19976191] (req 19972096->19976191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.010544:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19972096->19976191] (req 19972096->19976191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.010570:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19972096->19976191] (req 19972096->19976191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.010579:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f11 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19972096->19976191] (req 19972096->19976191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760725.010715:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+6:2424:x1876699484187008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39752us (40524us total) trans 4294972194 rc 0/0 00000100:00100000:3.0:1789760725.010728:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4905 00000100:00100000:3.0:1789760725.010739:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714265599242 time 714301872208 state 0x2 ref 0 00000100:00100000:0.0:1789760725.014692:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760725.014756:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484187136 00000100:00100000:3.0:1789760725.014770:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484187136 00000100:00100000:3.0:1789760725.014846:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714265599242 time 714301872208 state 0x0 ref 1 00000100:00100000:3.0:1789760725.014864:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714302132624 time 714301872208 state 0x1 ref 1 00000100:00100000:3.0:1789760725.014901:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714302132624 time 714345831324 state 0x0 ref 1 00000100:00100000:3.0:1789760725.014915:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484187136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760725.014980:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19976192, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:3.0:1789760725.014991:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.015011:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760725.015016:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760725.015023:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19976192->19980287] (req 19976192->19980287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.015040:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19976192->19980287] (req 19976192->19980287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760725.068111:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972195 is committed 00002000:00010000:0.0:1789760725.071823:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19976192 -> 19980288 00010000:00010000:0.0:1789760725.071854:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19976192->19980287] (req 19976192->19980287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.071874:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19976192->19980287] (req 19976192->19980287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.071884:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19976192->19980287] (req 19976192->19980287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.071893:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760725.071899:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19976192->19980287] (req 19976192->19980287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.071910:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19976192->19980287] (req 19976192->19980287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.071939:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19976192->19980287] (req 19976192->19980287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.071954:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f18 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19976192->19980287] (req 19976192->19980287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760725.072272:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484187136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57351us (57587us total) trans 4294972195 rc 0/0 00000100:00100000:0.0:1789760725.072287:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4906 00000100:00100000:0.0:1789760725.072300:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714302132624 time 714345831324 state 0x2 ref 0 00000100:00100000:1.0:1789760725.075805:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760725.076127:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484187264 00000100:00100000:2.0:1789760725.076141:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484187264 00000100:00100000:2.0:1789760725.076223:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714302132624 time 714345831324 state 0x0 ref 1 00000100:00100000:2.0:1789760725.076242:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714346091740 time 714345831324 state 0x1 ref 1 00000100:00100000:2.0:1789760725.076278:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714346091740 time 714407208862 state 0x0 ref 1 00000100:00100000:2.0:1789760725.076292:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484187264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760725.076358:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19980288, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:2.0:1789760725.076366:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.076387:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:2.0:1789760725.076391:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760725.076396:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19980288->19984383] (req 19980288->19984383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.076409:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19980288->19984383] (req 19980288->19984383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760725.131813:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972196 is committed 00002000:00010000:3.0:1789760725.132356:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19980288 -> 19984384 00010000:00010000:3.0:1789760725.132378:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19980288->19984383] (req 19980288->19984383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.132393:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19980288->19984383] (req 19980288->19984383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.132404:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19980288->19984383] (req 19980288->19984383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.132413:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760725.132420:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19980288->19984383] (req 19980288->19984383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.132430:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19980288->19984383] (req 19980288->19984383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.132466:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19980288->19984383] (req 19980288->19984383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.132475:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396207f1f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19980288->19984383] (req 19980288->19984383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760725.132608:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484187264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56312us (56808us total) trans 4294972196 rc 0/0 00000100:00100000:3.0:1789760725.132619:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4907 00000100:00100000:3.0:1789760725.132633:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714346091740 time 714407208862 state 0x2 ref 0 00000100:00100000:1.0:1789760725.139764:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760725.139868:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484187392 00000100:00100000:2.0:1789760725.139886:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484187392 00000100:00100000:2.0:1789760725.139979:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714346091740 time 714407208862 state 0x0 ref 1 00000100:00100000:2.0:1789760725.139999:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714407469278 time 714407208862 state 0x1 ref 1 00000100:00100000:2.0:1789760725.140039:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714407469278 time 714470969468 state 0x0 ref 1 00000100:00100000:2.0:1789760725.140054:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484187392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760725.140186:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19984384, mtime: 0x6aad94d4, atime: 0x0, ctime: 0x6aad94d4, blocks: 0xa007 00010000:00010000:2.0:1789760725.140198:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.140218:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:2.0:1789760725.140223:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760725.140231:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19984384->19988479] (req 19984384->19988479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.140247:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19984384->19988479] (req 19984384->19988479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760725.175875:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972197 is committed 00002000:00010000:2.0:1789760725.175985:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19984384 -> 19988480 00002000:00010000:2.0:1789760725.175990:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760724 -> 1789760725 00002000:00010000:2.0:1789760725.175993:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760724 -> 1789760725 00010000:00010000:2.0:1789760725.176008:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19984384->19988479] (req 19984384->19988479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.176021:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19984384->19988479] (req 19984384->19988479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.176030:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19984384->19988479] (req 19984384->19988479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.176038:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760725.176044:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19984384->19988479] (req 19984384->19988479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.176053:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19984384->19988479] (req 19984384->19988479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.176165:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19984384->19988479] (req 19984384->19988479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.176174:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f26 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19984384->19988479] (req 19984384->19988479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760725.176268:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484187392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36213us (36510us total) trans 4294972197 rc 0/0 00000100:00100000:2.0:1789760725.176278:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4908 00000100:00100000:2.0:1789760725.176288:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714407469278 time 714470969468 state 0x2 ref 0 00000100:00100000:2.0:1789760725.180367:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760725.180527:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484187520 00000100:00100000:3.0:1789760725.180598:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484187520 00000100:00100000:3.0:1789760725.180659:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714407469278 time 714470969468 state 0x0 ref 1 00000100:00100000:3.0:1789760725.180676:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714471229884 time 714470969468 state 0x1 ref 1 00000100:00100000:3.0:1789760725.180737:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714471229884 time 714511667156 state 0x0 ref 1 00000100:00100000:3.0:1789760725.180750:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484187520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760725.180800:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19988480, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:3.0:1789760725.180810:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.180826:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760725.180831:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760725.180838:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19988480->19992575] (req 19988480->19992575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.180853:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19988480->19992575] (req 19988480->19992575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760725.212512:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972198 is committed 00002000:00010000:2.0:1789760725.212808:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19988480 -> 19992576 00010000:00010000:2.0:1789760725.212824:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19988480->19992575] (req 19988480->19992575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.212838:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19988480->19992575] (req 19988480->19992575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.212847:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19988480->19992575] (req 19988480->19992575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.212861:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760725.212866:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19988480->19992575] (req 19988480->19992575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.212875:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19988480->19992575] (req 19988480->19992575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.212893:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19988480->19992575] (req 19988480->19992575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.212900:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396207f2d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19988480->19992575] (req 19988480->19992575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760725.212985:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484187520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32233us (32624us total) trans 4294972198 rc 0/0 00000100:00100000:2.0:1789760725.212996:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4909 00000100:00100000:2.0:1789760725.213005:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714471229884 time 714511667156 state 0x2 ref 0 00000100:00100000:1.0:1789760725.217018:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760725.218847:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484187648 00000100:00100000:3.0:1789760725.218862:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484187648 00000100:00100000:3.0:1789760725.218923:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714471229884 time 714511667156 state 0x0 ref 1 00000100:00100000:3.0:1789760725.218938:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714511927572 time 714511667156 state 0x1 ref 1 00000100:00100000:3.0:1789760725.218997:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714511927572 time 714549928896 state 0x0 ref 1 00000100:00100000:3.0:1789760725.219008:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484187648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760725.219050:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19992576, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:3.0:1789760725.219058:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.219106:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760725.219109:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760725.219113:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19992576->19996671] (req 19992576->19996671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.219124:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19992576->19996671] (req 19992576->19996671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760725.256781:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972199 is committed 00002000:00010000:3.0:1789760725.256883:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19992576 -> 19996672 00010000:00010000:3.0:1789760725.256900:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19992576->19996671] (req 19992576->19996671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.256912:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19992576->19996671] (req 19992576->19996671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.256920:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19992576->19996671] (req 19992576->19996671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.256926:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760725.256931:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19992576->19996671] (req 19992576->19996671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.256939:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19992576->19996671] (req 19992576->19996671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.256958:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19992576->19996671] (req 19992576->19996671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.256967:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f34 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19992576->19996671] (req 19992576->19996671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760725.257055:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484187648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38042us (40041us total) trans 4294972199 rc 0/0 00000100:00100000:3.0:1789760725.257947:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4910 00000100:00100000:3.0:1789760725.257959:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714511927572 time 714549928896 state 0x2 ref 0 00000100:00100000:1.0:1789760725.266608:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760725.267251:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484187776 00000100:00100000:2.0:1789760725.267262:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484187776 00000100:00100000:2.0:1789760725.267319:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714511927572 time 714549928896 state 0x0 ref 1 00000100:00100000:2.0:1789760725.267342:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714550189312 time 714549928896 state 0x1 ref 1 00000100:00100000:2.0:1789760725.267373:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714550189312 time 714598304614 state 0x0 ref 1 00000100:00100000:2.0:1789760725.267388:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484187776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760725.267446:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 19996672, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:2.0:1789760725.267456:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.267474:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:2.0:1789760725.267480:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760725.267488:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19996672->20000767] (req 19996672->20000767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.267506:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19996672->20000767] (req 19996672->20000767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760725.327339:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972200 is committed 00002000:00010000:3.0:1789760725.327603:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 19996672 -> 20000768 00010000:00010000:3.0:1789760725.327619:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19996672->20000767] (req 19996672->20000767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.327633:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19996672->20000767] (req 19996672->20000767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.327643:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19996672->20000767] (req 19996672->20000767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.327651:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760725.327656:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19996672->20000767] (req 19996672->20000767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.327666:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19996672->20000767] (req 19996672->20000767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.327693:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19996672->20000767] (req 19996672->20000767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.327701:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396207f3b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [19996672->20000767] (req 19996672->20000767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760725.327837:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484187776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60448us (61397us total) trans 4294972200 rc 0/0 00000100:00100000:3.0:1789760725.327848:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4911 00000100:00100000:3.0:1789760725.327858:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714550189312 time 714598304614 state 0x2 ref 0 00000100:00100000:2.0:1789760725.332628:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760725.332724:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484187904 00000100:00100000:0.0:1789760725.332735:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484187904 00000100:00100000:0.0:1789760725.332795:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714550189312 time 714598304614 state 0x0 ref 1 00000100:00100000:0.0:1789760725.332811:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714598565030 time 714598304614 state 0x1 ref 1 00000100:00100000:0.0:1789760725.332842:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714598565030 time 714663773421 state 0x0 ref 1 00000100:00100000:0.0:1789760725.332856:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484187904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760725.332902:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20000768, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:0.0:1789760725.332911:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.332928:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:0.0:1789760725.332932:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760725.332937:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20000768->20004863] (req 20000768->20004863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.332956:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20000768->20004863] (req 20000768->20004863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760725.400368:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20000768 -> 20004864 00010000:00010000:1.0:1789760725.400409:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20000768->20004863] (req 20000768->20004863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.400425:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20000768->20004863] (req 20000768->20004863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.400442:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20000768->20004863] (req 20000768->20004863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.400451:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760725.400457:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20000768->20004863] (req 20000768->20004863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.400468:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20000768->20004863] (req 20000768->20004863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.400499:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20000768->20004863] (req 20000768->20004863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.400507:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396207f42 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20000768->20004863] (req 20000768->20004863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760725.400643:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484187904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67783us (68017us total) trans 4294972201 rc 0/0 00000100:00100000:1.0:1789760725.400656:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4912 00000100:00100000:1.0:1789760725.400669:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714598565030 time 714663773421 state 0x2 ref 0 00000001:00080000:3.0:1789760725.400957:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972201 is committed 00000100:00100000:1.0:1789760725.404255:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760725.404346:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484188032 00000100:00100000:3.0:1789760725.404359:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484188032 00000100:00100000:3.0:1789760725.404444:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714598565030 time 714663773421 state 0x0 ref 1 00000100:00100000:3.0:1789760725.404486:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714664033837 time 714663773421 state 0x1 ref 1 00000100:00100000:3.0:1789760725.404524:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714664033837 time 714735454753 state 0x0 ref 1 00000100:00100000:3.0:1789760725.404546:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484188032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760725.404613:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20004864, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:3.0:1789760725.404624:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.404648:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760725.404654:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760725.404663:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20004864->20008959] (req 20004864->20008959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.404679:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20004864->20008959] (req 20004864->20008959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760725.451954:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972202 is committed 00002000:00010000:0.0:1789760725.452334:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20004864 -> 20008960 00010000:00010000:0.0:1789760725.452355:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20004864->20008959] (req 20004864->20008959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.452371:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20004864->20008959] (req 20004864->20008959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.452382:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20004864->20008959] (req 20004864->20008959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.452391:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760725.452397:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20004864->20008959] (req 20004864->20008959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.452408:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20004864->20008959] (req 20004864->20008959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.452445:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20004864->20008959] (req 20004864->20008959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.452454:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396207f49 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20004864->20008959] (req 20004864->20008959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760725.452809:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484188032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48258us (48560us total) trans 4294972202 rc 0/0 00000100:00100000:0.0:1789760725.452821:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4913 00000100:00100000:0.0:1789760725.452832:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714664033837 time 714735454753 state 0x2 ref 0 00000100:00100000:2.0:1789760725.460383:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760725.460510:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484188160 00000100:00100000:3.0:1789760725.460534:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484188160 00000100:00100000:3.0:1789760725.460607:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714664033837 time 714735454753 state 0x0 ref 1 00000100:00100000:3.0:1789760725.460627:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714735715169 time 714735454753 state 0x1 ref 1 00000100:00100000:3.0:1789760725.460666:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714735715169 time 714791596164 state 0x0 ref 1 00000100:00100000:3.0:1789760725.460680:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484188160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760725.460750:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20008960, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:3.0:1789760725.460761:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.460782:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760725.460787:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760725.460794:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20008960->20013055] (req 20008960->20013055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.460817:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20008960->20013055] (req 20008960->20013055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760725.508743:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20008960 -> 20013056 00010000:00010000:2.0:1789760725.508772:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20008960->20013055] (req 20008960->20013055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.508787:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20008960->20013055] (req 20008960->20013055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.508796:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20008960->20013055] (req 20008960->20013055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.508804:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760725.508810:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20008960->20013055] (req 20008960->20013055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.508821:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20008960->20013055] (req 20008960->20013055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.508846:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20008960->20013055] (req 20008960->20013055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.508855:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396207f50 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20008960->20013055] (req 20008960->20013055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760725.508981:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484188160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48296us (48607us total) trans 4294972203 rc 0/0 00000100:00100000:2.0:1789760725.508993:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4914 00000100:00100000:2.0:1789760725.509004:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714735715169 time 714791596164 state 0x2 ref 0 00000001:00080000:0.0:1789760725.510479:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972203 is committed 00000100:00100000:0.0:1789760725.512034:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760725.512485:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484188288 00000100:00100000:3.0:1789760725.512500:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484188288 00000100:00100000:3.0:1789760725.512581:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714735715169 time 714791596164 state 0x0 ref 1 00000100:00100000:3.0:1789760725.512602:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714791856580 time 714791596164 state 0x1 ref 1 00000100:00100000:3.0:1789760725.512639:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714791856580 time 714843569538 state 0x0 ref 1 00000100:00100000:3.0:1789760725.512655:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484188288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760725.512723:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20013056, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:3.0:1789760725.512734:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.512755:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760725.512760:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760725.512768:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20013056->20017151] (req 20013056->20017151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.512783:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20013056->20017151] (req 20013056->20017151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760725.547058:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20013056 -> 20017152 00010000:00010000:0.0:1789760725.547118:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20013056->20017151] (req 20013056->20017151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.547134:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20013056->20017151] (req 20013056->20017151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.547144:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20013056->20017151] (req 20013056->20017151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.547152:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760725.547157:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20013056->20017151] (req 20013056->20017151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.547166:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20013056->20017151] (req 20013056->20017151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.547190:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20013056->20017151] (req 20013056->20017151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.547198:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207f57 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20013056->20017151] (req 20013056->20017151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760725.547327:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484188288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34669us (35300us total) trans 4294972204 rc 0/0 00000100:00100000:0.0:1789760725.547340:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4915 00000100:00100000:0.0:1789760725.547353:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714791856580 time 714843569538 state 0x2 ref 0 00000001:00080000:0.0:1789760725.547408:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972204 is committed 00000100:00100000:3.0:1789760725.553133:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760725.553513:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484188416 00000100:00100000:1.0:1789760725.553590:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484188416 00000100:00100000:1.0:1789760725.553671:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714791856580 time 714843569538 state 0x0 ref 1 00000100:00100000:1.0:1789760725.553692:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714843829954 time 714843569538 state 0x1 ref 1 00000100:00100000:1.0:1789760725.553730:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714843829954 time 714884660055 state 0x0 ref 1 00000100:00100000:1.0:1789760725.553745:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484188416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760725.553812:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20017152, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:1.0:1789760725.553825:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.553850:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:1.0:1789760725.553856:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760725.553864:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20017152->20021247] (req 20017152->20021247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.553881:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20017152->20021247] (req 20017152->20021247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760725.593262:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972205 is committed 00002000:00010000:1.0:1789760725.593454:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20017152 -> 20021248 00010000:00010000:1.0:1789760725.593474:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20017152->20021247] (req 20017152->20021247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.593490:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20017152->20021247] (req 20017152->20021247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.593501:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20017152->20021247] (req 20017152->20021247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.593510:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760725.593516:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20017152->20021247] (req 20017152->20021247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.593527:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20017152->20021247] (req 20017152->20021247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.593552:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20017152->20021247] (req 20017152->20021247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.593561:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396207f5e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20017152->20021247] (req 20017152->20021247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760725.593672:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484188416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39924us (40548us total) trans 4294972205 rc 0/0 00000100:00100000:1.0:1789760725.593685:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4916 00000100:00100000:1.0:1789760725.593697:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714843829954 time 714884660055 state 0x2 ref 0 00000100:00100000:1.0:1789760725.596232:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760725.596328:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484188544 00000100:00100000:0.0:1789760725.596342:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484188544 00000100:00100000:0.0:1789760725.596429:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714843829954 time 714884660055 state 0x0 ref 1 00000100:00100000:0.0:1789760725.596459:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714884920471 time 714884660055 state 0x1 ref 1 00000100:00100000:0.0:1789760725.596548:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714884920471 time 714927477206 state 0x0 ref 1 00000100:00100000:0.0:1789760725.596565:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484188544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760725.596638:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20021248, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:0.0:1789760725.596649:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.596670:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:0.0:1789760725.596675:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760725.596682:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20021248->20025343] (req 20021248->20025343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.596700:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20021248->20025343] (req 20021248->20025343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760725.639505:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972206 is committed 00002000:00010000:0.0:1789760725.639776:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20021248 -> 20025344 00010000:00010000:0.0:1789760725.639796:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20021248->20025343] (req 20021248->20025343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.639812:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20021248->20025343] (req 20021248->20025343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.639822:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20021248->20025343] (req 20021248->20025343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.639830:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760725.639836:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20021248->20025343] (req 20021248->20025343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.639846:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20021248->20025343] (req 20021248->20025343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.639871:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20021248->20025343] (req 20021248->20025343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.639879:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207f65 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20021248->20025343] (req 20021248->20025343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760725.640010:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484188544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43444us (43787us total) trans 4294972206 rc 0/0 00000100:00100000:0.0:1789760725.640022:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4917 00000100:00100000:0.0:1789760725.640033:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714884920471 time 714927477206 state 0x2 ref 0 00000100:00100000:0.0:1789760725.644377:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760725.644643:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484188672 00000100:00100000:3.0:1789760725.644661:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484188672 00000100:00100000:3.0:1789760725.644739:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714884920471 time 714927477206 state 0x0 ref 1 00000100:00100000:3.0:1789760725.644758:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714927737622 time 714927477206 state 0x1 ref 1 00000100:00100000:3.0:1789760725.644804:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714927737622 time 714975735017 state 0x0 ref 1 00000100:00100000:3.0:1789760725.644820:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484188672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760725.644889:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20025344, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:3.0:1789760725.644901:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.644928:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=714s v=(0 0) 00010000:00010000:3.0:1789760725.644933:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760725.644942:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20025344->20029439] (req 20025344->20029439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.644960:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20025344->20029439] (req 20025344->20029439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760725.681025:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20025344 -> 20029440 00010000:00010000:3.0:1789760725.681055:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20025344->20029439] (req 20025344->20029439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.681107:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20025344->20029439] (req 20025344->20029439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.681120:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20025344->20029439] (req 20025344->20029439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.681129:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760725.681135:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20025344->20029439] (req 20025344->20029439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.681146:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20025344->20029439] (req 20025344->20029439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.681174:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20025344->20029439] (req 20025344->20029439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.681184:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f6c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20025344->20029439] (req 20025344->20029439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760725.681309:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484188672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36487us (36941us total) trans 4294972207 rc 0/0 00000100:00100000:3.0:1789760725.681323:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4918 00000100:00100000:3.0:1789760725.681336:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714927737622 time 714975735017 state 0x2 ref 0 00000001:00080000:1.0:1789760725.681993:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972207 is committed 00000100:00100000:3.0:1789760725.683902:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760725.684379:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484188800 00000100:00100000:0.0:1789760725.684393:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484188800 00000100:00100000:0.0:1789760725.684491:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714927737622 time 714975735017 state 0x0 ref 1 00000100:00100000:0.0:1789760725.684510:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714975995433 time 714975735017 state 0x1 ref 1 00000100:00100000:0.0:1789760725.684551:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714975995433 time 715015481430 state 0x0 ref 1 00000100:00100000:0.0:1789760725.684566:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484188800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760725.684630:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20029440, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:0.0:1789760725.684640:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.684659:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:0.0:1789760725.684665:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760725.684673:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20029440->20033535] (req 20029440->20033535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.684689:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20029440->20033535] (req 20029440->20033535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760725.713246:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972208 is committed 00002000:00010000:0.0:1789760725.713516:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20029440 -> 20033536 00010000:00010000:0.0:1789760725.713601:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20029440->20033535] (req 20029440->20033535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.713617:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20029440->20033535] (req 20029440->20033535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.713627:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20029440->20033535] (req 20029440->20033535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.713635:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760725.713641:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20029440->20033535] (req 20029440->20033535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.713652:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20029440->20033535] (req 20029440->20033535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.713677:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20029440->20033535] (req 20029440->20033535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.713686:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396207f73 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20029440->20033535] (req 20029440->20033535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760725.713867:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484188800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29296us (29969us total) trans 4294972208 rc 0/0 00000100:00100000:0.0:1789760725.713883:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4919 00000100:00100000:0.0:1789760725.713895:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714975995433 time 715015481430 state 0x2 ref 0 00000100:00100000:0.0:1789760725.721874:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760725.721992:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484188928 00000100:00100000:1.0:1789760725.722016:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484188928 00000100:00100000:1.0:1789760725.722140:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 714975995433 time 715015481430 state 0x0 ref 1 00000100:00100000:1.0:1789760725.722171:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715015741846 time 715015481430 state 0x1 ref 1 00000100:00100000:3.0:1789760725.722245:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715015741846 time 715053172165 state 0x0 ref 1 00000100:00100000:3.0:1789760725.722269:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484188928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760725.722339:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20033536, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:3.0:1789760725.722351:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.722376:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:3.0:1789760725.722381:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760725.722390:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20033536->20037631] (req 20033536->20037631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.722408:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20033536->20037631] (req 20033536->20037631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760725.763660:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972209 is committed 00002000:00010000:0.0:1789760725.763767:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20033536 -> 20037632 00010000:00010000:0.0:1789760725.763789:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20033536->20037631] (req 20033536->20037631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.763804:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20033536->20037631] (req 20033536->20037631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.763815:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20033536->20037631] (req 20033536->20037631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.763824:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760725.763830:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20033536->20037631] (req 20033536->20037631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.763841:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20033536->20037631] (req 20033536->20037631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.763865:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20033536->20037631] (req 20033536->20037631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.763874:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396207f7a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20033536->20037631] (req 20033536->20037631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760725.764039:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484188928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41771us (42291us total) trans 4294972209 rc 0/0 00000100:00100000:0.0:1789760725.764052:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4920 00000100:00100000:0.0:1789760725.764108:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715015741846 time 715053172165 state 0x2 ref 0 00000100:00100000:2.0:1789760725.772885:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760725.772989:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484189056 00000100:00100000:3.0:1789760725.773002:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484189056 00000100:00100000:3.0:1789760725.773187:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715015741846 time 715053172165 state 0x0 ref 1 00000100:00100000:3.0:1789760725.773207:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715053432581 time 715053172165 state 0x1 ref 1 00000100:00100000:2.0:1789760725.776569:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715053432581 time 715107479453 state 0x0 ref 1 00000100:00100000:2.0:1789760725.776613:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760725.776693:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20037632, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:2.0:1789760725.776705:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.776730:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:2.0:1789760725.776735:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760725.776743:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20037632->20041727] (req 20037632->20041727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.776759:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20037632->20041727] (req 20037632->20041727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760725.807337:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972210 is committed 00002000:00010000:2.0:1789760725.807509:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20037632 -> 20041728 00010000:00010000:2.0:1789760725.807531:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20037632->20041727] (req 20037632->20041727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.807546:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20037632->20041727] (req 20037632->20041727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.807556:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20037632->20041727] (req 20037632->20041727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.807565:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760725.807571:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20037632->20041727] (req 20037632->20041727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.807580:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20037632->20041727] (req 20037632->20041727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.807604:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20037632->20041727] (req 20037632->20041727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760725.807614:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396207f81 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20037632->20041727] (req 20037632->20041727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760725.807772:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31172us (34894us total) trans 4294972210 rc 0/0 00000100:00100000:2.0:1789760725.807785:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4921 00000100:00100000:2.0:1789760725.807799:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715053432581 time 715107479453 state 0x2 ref 0 00000100:00100000:3.0:1789760725.810776:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760725.811060:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484189184 00000100:00100000:0.0:1789760725.811111:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484189184 00000100:00100000:0.0:1789760725.811191:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715053432581 time 715107479453 state 0x0 ref 1 00000100:00100000:0.0:1789760725.811209:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715107739869 time 715107479453 state 0x1 ref 1 00000100:00100000:0.0:1789760725.811254:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715107739869 time 715142184248 state 0x0 ref 1 00000100:00100000:0.0:1789760725.811270:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484189184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760725.811344:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20041728, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:0.0:1789760725.811354:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.811374:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:0.0:1789760725.811379:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760725.811387:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20041728->20045823] (req 20041728->20045823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.811403:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20041728->20045823] (req 20041728->20045823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760725.849311:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20041728 -> 20045824 00010000:00010000:0.0:1789760725.849341:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20041728->20045823] (req 20041728->20045823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.849357:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20041728->20045823] (req 20041728->20045823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.849368:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20041728->20045823] (req 20041728->20045823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.849376:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760725.849382:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20041728->20045823] (req 20041728->20045823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.849393:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20041728->20045823] (req 20041728->20045823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.849420:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20041728->20045823] (req 20041728->20045823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.849428:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f88 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20041728->20045823] (req 20041728->20045823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760725.849578:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484189184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38304us (38809us total) trans 4294972211 rc 0/0 00000100:00100000:0.0:1789760725.849594:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4922 00000100:00100000:0.0:1789760725.849607:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715107739869 time 715142184248 state 0x2 ref 0 00000001:00080000:2.0:1789760725.849816:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972211 is committed 00000100:00100000:0.0:1789760725.856198:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760725.856291:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484189312 00000100:00100000:3.0:1789760725.856305:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484189312 00000100:00100000:3.0:1789760725.856384:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715107739869 time 715142184248 state 0x0 ref 1 00000100:00100000:3.0:1789760725.856404:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715142444664 time 715142184248 state 0x1 ref 1 00000100:00100000:3.0:1789760725.856486:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715142444664 time 715187416067 state 0x0 ref 1 00000100:00100000:3.0:1789760725.856500:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760725.856569:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20045824, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:3.0:1789760725.856579:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.856601:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:3.0:1789760725.856606:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760725.856613:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20045824->20049919] (req 20045824->20049919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760725.856631:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20045824->20049919] (req 20045824->20049919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760725.900546:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760725.949244:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972212 is committed 00002000:00010000:0.0:1789760725.949338:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20045824 -> 20049920 00010000:00010000:0.0:1789760725.949359:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20045824->20049919] (req 20045824->20049919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.949374:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20045824->20049919] (req 20045824->20049919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.949384:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20045824->20049919] (req 20045824->20049919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.949393:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760725.949399:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20045824->20049919] (req 20045824->20049919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.949410:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20045824->20049919] (req 20045824->20049919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.949446:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20045824->20049919] (req 20045824->20049919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.949455:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207f8f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20045824->20049919] (req 20045824->20049919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760725.949704:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 93198us (93511us total) trans 4294972212 rc 0/0 00000100:00100000:0.0:1789760725.949719:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4923 00000100:00100000:0.0:1789760725.949731:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715142444664 time 715187416067 state 0x2 ref 0 00000100:00100000:3.0:1789760725.954575:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760725.954639:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484189440 00000100:00100000:0.0:1789760725.954652:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484189440 00000100:00100000:0.0:1789760725.954738:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715142444664 time 715187416067 state 0x0 ref 1 00000100:00100000:0.0:1789760725.954769:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715187676483 time 715187416067 state 0x1 ref 1 00000100:00100000:0.0:1789760725.954809:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715187676483 time 715285738821 state 0x0 ref 1 00000100:00100000:0.0:1789760725.954825:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760725.954893:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20049920, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:0.0:1789760725.954905:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.954925:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:0.0:1789760725.954930:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760725.954939:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20049920->20054015] (req 20049920->20054015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760725.954955:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20049920->20054015] (req 20049920->20054015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760726.035803:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972213 is committed 00002000:00010000:0.0:1789760726.036150:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20049920 -> 20054016 00010000:00010000:0.0:1789760726.036170:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20049920->20054015] (req 20049920->20054015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.036186:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20049920->20054015] (req 20049920->20054015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.036197:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20049920->20054015] (req 20049920->20054015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.036205:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760726.036212:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20049920->20054015] (req 20049920->20054015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.036223:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20049920->20054015] (req 20049920->20054015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.036251:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20049920->20054015] (req 20049920->20054015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.036259:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f96 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20049920->20054015] (req 20049920->20054015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760726.036381:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 81554us (81815us total) trans 4294972213 rc 0/0 00000100:00100000:0.0:1789760726.036396:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4924 00000100:00100000:0.0:1789760726.036420:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715187676483 time 715285738821 state 0x2 ref 0 00000100:00100000:2.0:1789760726.042545:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760726.042645:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484189568 00000100:00100000:0.0:1789760726.042658:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484189568 00000100:00100000:0.0:1789760726.042768:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715187676483 time 715285738821 state 0x0 ref 1 00000100:00080000:3.0:1789760726.042777:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760726.042787:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715285999237 time 715285738821 state 0x1 ref 1 00000100:00100000:0.0:1789760726.042825:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715285999237 time 715373754979 state 0x0 ref 1 00000100:00100000:0.0:1789760726.042840:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484189568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760726.042903:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20054016, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:0.0:1789760726.042913:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.042933:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:0.0:1789760726.042938:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760726.042946:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20054016->20058111] (req 20054016->20058111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.042963:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20054016->20058111] (req 20054016->20058111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760726.101098:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972214 is committed 00002000:00010000:1.0:1789760726.101509:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20054016 -> 20058112 00010000:00010000:1.0:1789760726.101530:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20054016->20058111] (req 20054016->20058111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.101545:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20054016->20058111] (req 20054016->20058111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.101557:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20054016->20058111] (req 20054016->20058111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.101565:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760726.101571:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20054016->20058111] (req 20054016->20058111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.101581:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20054016->20058111] (req 20054016->20058111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.101606:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20054016->20058111] (req 20054016->20058111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.101613:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396207f9d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20054016->20058111] (req 20054016->20058111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760726.101769:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484189568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58925us (59230us total) trans 4294972214 rc 0/0 00000100:00100000:1.0:1789760726.101781:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4925 00000100:00100000:1.0:1789760726.101793:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715285999237 time 715373754979 state 0x2 ref 0 00000100:00100000:1.0:1789760726.107352:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760726.107674:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484189696 00000100:00100000:2.0:1789760726.107688:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484189696 00000100:00100000:2.0:1789760726.107776:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715285999237 time 715373754979 state 0x0 ref 1 00000100:00100000:2.0:1789760726.107795:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715374015395 time 715373754979 state 0x1 ref 1 00000100:00100000:2.0:1789760726.107831:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715374015395 time 715438761514 state 0x0 ref 1 00000100:00100000:2.0:1789760726.107845:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760726.107910:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20058112, mtime: 0x6aad94d5, atime: 0x0, ctime: 0x6aad94d5, blocks: 0xa007 00010000:00010000:2.0:1789760726.107921:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.107941:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:2.0:1789760726.107946:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760726.107954:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20058112->20062207] (req 20058112->20062207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.107970:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20058112->20062207] (req 20058112->20062207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760726.173427:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972215 is committed 00002000:00010000:2.0:1789760726.174163:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20058112 -> 20062208 00002000:00010000:2.0:1789760726.174170:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760725 -> 1789760726 00002000:00010000:2.0:1789760726.174173:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760725 -> 1789760726 00010000:00010000:2.0:1789760726.174196:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20058112->20062207] (req 20058112->20062207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.174336:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20058112->20062207] (req 20058112->20062207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.174348:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20058112->20062207] (req 20058112->20062207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.174355:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760726.174361:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20058112->20062207] (req 20058112->20062207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.174373:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20058112->20062207] (req 20058112->20062207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.174400:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20058112->20062207] (req 20058112->20062207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.174414:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fa4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20058112->20062207] (req 20058112->20062207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760726.174628:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66778us (67283us total) trans 4294972215 rc 0/0 00000100:00100000:3.0:1789760726.174642:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4926 00000100:00100000:3.0:1789760726.174656:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715374015395 time 715438761514 state 0x2 ref 0 00000100:00100000:2.0:1789760726.177863:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760726.178016:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484189824 00000100:00100000:3.0:1789760726.178031:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484189824 00000100:00100000:3.0:1789760726.178138:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715374015395 time 715438761514 state 0x0 ref 1 00000100:00100000:3.0:1789760726.178157:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715439021930 time 715438761514 state 0x1 ref 1 00000100:00100000:3.0:1789760726.178186:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715439021930 time 715509118015 state 0x0 ref 1 00000100:00100000:3.0:1789760726.178201:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7072050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760726.178276:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20062208, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:3.0:1789760726.178286:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.178306:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:3.0:1789760726.178310:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760726.178316:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20062208->20066303] (req 20062208->20066303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.178331:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20062208->20066303] (req 20062208->20066303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760726.241910:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20062208 -> 20066304 00010000:00010000:1.0:1789760726.241939:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20062208->20066303] (req 20062208->20066303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.241957:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20062208->20066303] (req 20062208->20066303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.241968:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20062208->20066303] (req 20062208->20066303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.241977:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760726.241983:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20062208->20066303] (req 20062208->20066303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.241995:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20062208->20066303] (req 20062208->20066303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.242029:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20062208->20066303] (req 20062208->20066303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.242038:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396207fab lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20062208->20066303] (req 20062208->20066303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760726.242204:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7072050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484189824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63996us (64349us total) trans 4294972216 rc 0/0 00000100:00100000:1.0:1789760726.242216:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4927 00000100:00100000:1.0:1789760726.242229:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715439021930 time 715509118015 state 0x2 ref 0 00000001:00080000:2.0:1789760726.243837:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972216 is committed 00000100:00100000:2.0:1789760726.246600:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760726.246750:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484189952 00000100:00100000:3.0:1789760726.246764:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484189952 00000100:00100000:3.0:1789760726.246929:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715439021930 time 715509118015 state 0x0 ref 1 00000100:00100000:3.0:1789760726.246949:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715509378431 time 715509118015 state 0x1 ref 1 00000100:00100000:3.0:1789760726.246986:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715509378431 time 715577916842 state 0x0 ref 1 00000100:00100000:3.0:1789760726.247002:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484189952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760726.247098:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20066304, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:3.0:1789760726.247110:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.247136:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:3.0:1789760726.247141:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760726.247148:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20066304->20070399] (req 20066304->20070399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.247164:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20066304->20070399] (req 20066304->20070399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760726.293630:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20066304 -> 20070400 00010000:00010000:1.0:1789760726.293660:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20066304->20070399] (req 20066304->20070399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.293678:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20066304->20070399] (req 20066304->20070399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.293688:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20066304->20070399] (req 20066304->20070399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.293697:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760726.293704:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20066304->20070399] (req 20066304->20070399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.293716:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20066304->20070399] (req 20066304->20070399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.293740:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20066304->20070399] (req 20066304->20070399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.293748:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396207fb2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20066304->20070399] (req 20066304->20070399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760726.293884:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484189952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46878us (47293us total) trans 4294972217 rc 0/0 00000100:00100000:1.0:1789760726.293897:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4928 00000100:00100000:1.0:1789760726.293910:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715509378431 time 715577916842 state 0x2 ref 0 00000100:00100000:1.0:1789760726.296328:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760726.296463:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484190080 00000100:00100000:0.0:1789760726.296478:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484190080 00000100:00100000:0.0:1789760726.296572:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715509378431 time 715577916842 state 0x0 ref 1 00000100:00100000:0.0:1789760726.296592:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715578177258 time 715577916842 state 0x1 ref 1 00000100:00100000:0.0:1789760726.296628:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715578177258 time 715627558130 state 0x0 ref 1 00000100:00100000:0.0:1789760726.296645:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484190080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760726.296714:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20070400, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:0.0:1789760726.296725:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.296747:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:0.0:1789760726.296752:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760726.296759:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20070400->20074495] (req 20070400->20074495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.296777:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20070400->20074495] (req 20070400->20074495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760726.298147:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972217 is committed 00000001:00080000:0.0:1789760726.361820:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972218 is committed 00002000:00010000:1.0:1789760726.361954:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20070400 -> 20074496 00010000:00010000:1.0:1789760726.361974:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20070400->20074495] (req 20070400->20074495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.361989:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20070400->20074495] (req 20070400->20074495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.362000:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20070400->20074495] (req 20070400->20074495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.362008:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760726.362014:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20070400->20074495] (req 20070400->20074495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.362026:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20070400->20074495] (req 20070400->20074495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.362050:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20070400->20074495] (req 20070400->20074495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.362058:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fb9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20070400->20074495] (req 20070400->20074495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760726.362271:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484190080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65564us (65889us total) trans 4294972218 rc 0/0 00000100:00100000:1.0:1789760726.362290:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4929 00000100:00100000:1.0:1789760726.362302:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715578177258 time 715627558130 state 0x2 ref 0 00000100:00100000:0.0:1789760726.365125:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760726.365264:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484190208 00000100:00100000:2.0:1789760726.365277:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484190208 00000100:00100000:2.0:1789760726.365350:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715578177258 time 715627558130 state 0x0 ref 1 00000100:00100000:2.0:1789760726.365368:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715627818546 time 715627558130 state 0x1 ref 1 00000100:00100000:2.0:1789760726.365401:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715627818546 time 715696331212 state 0x0 ref 1 00000100:00100000:2.0:1789760726.365415:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484190208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760726.365492:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20074496, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:2.0:1789760726.365503:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.365523:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:2.0:1789760726.365528:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760726.365536:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20074496->20078591] (req 20074496->20078591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.365551:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20074496->20078591] (req 20074496->20078591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760726.412325:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c12a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508913408:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760726.412436:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760726.412524:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760726.412532:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c10700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508913536:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760726.412557:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760726.412802:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5083100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3480:1876699508913664:0@lo:400:kworker.0 00000100:00100000:3.0:1789760726.412854:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760726.412880:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5081880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508914176:0@lo:400:kworker.0 00000100:00100000:3.0:1789760726.412908:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760726.412928:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5083b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508914688:0@lo:400:kworker.0 00000100:00100000:3.0:1789760726.412966:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760726.412988:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [3->2] 00000100:00100000:3.0:1789760726.412994:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5081180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508913792:0@lo:400:kworker.0 00000100:00100000:3.0:1789760726.413022:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760726.413040:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5082a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508914304:0@lo:400:kworker.0 00000100:00100000:3.0:1789760726.413062:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760726.413111:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5082d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508914816:0@lo:400:kworker.0 00000100:00100000:3.0:1789760726.413134:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760726.413167:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508914688 00000100:00100000:3.0:1789760726.413177:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508914688 00000100:00100000:3.0:1789760726.413244:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 312 00000100:00100000:3.0:1789760726.413256:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508914688:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760726.413317:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508914688:12345-0@lo:400:kworker.0 Request processed in 63us (349us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760726.413325:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 312 00000100:00080000:3.0:1789760726.413360:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760726.413399:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5083b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508914688:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.413696:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508913408 00000100:00100000:2.0:1789760726.413701:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508913408 00000100:00100000:2.0:1789760726.413754:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508913536 00000100:00100000:2.0:1789760726.413758:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508913536 00000100:00100000:2.0:1789760726.413782:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 368 00000100:00100000:2.0:1789760726.413788:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508913408:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1789760726.413825:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760726.413881:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508913408:12345-0@lo:13:osp-pre-1-0.0 Request processed in 92us (1457us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.413887:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 368 00000100:00100000:2.0:1789760726.413901:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 369 00000100:00100000:2.0:1789760726.413908:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508913536:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:2.0:1789760726.413918:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760726.413943:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508913536:12345-0@lo:13:osp-pre-1-1.0 Request processed in 36us (1385us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.413960:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 369 00000100:00100000:2.0:1789760726.413991:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5082300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508914048:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414019:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760726.414042:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5083800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508914560:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414081:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760726.414103:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1789760726.414109:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5080000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508913920:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414135:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760726.414153:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5083480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0000_UUID:3479:1876699508914432:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414173:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760726.414191:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5081500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0001_UUID:3479:1876699508914944:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414211:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760726.414238:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508913792 00000100:00100000:2.0:1789760726.414243:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508913792 00000100:00100000:2.0:1789760726.414268:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508914304 00000100:00100000:2.0:1789760726.414271:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508914304 00000100:00100000:2.0:1789760726.414285:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508914816 00000100:00100000:2.0:1789760726.414287:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508914816 00000100:00100000:2.0:1789760726.414299:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508914048 00000100:00100000:2.0:1789760726.414301:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508914048 00000100:00100000:2.0:1789760726.414312:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508914432 00000100:00100000:2.0:1789760726.414314:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508914432 00000100:00100000:2.0:1789760726.414325:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508914944 00000100:00100000:2.0:1789760726.414328:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508914944 00000100:00100000:2.0:1789760726.414341:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 885 00000100:00100000:2.0:1789760726.414347:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508913792:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414385:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508913792:12345-0@lo:400:kworker.0 Request processed in 37us (1363us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.414393:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 885 00000100:00100000:2.0:1789760726.414405:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 886 00000100:00100000:2.0:1789760726.414412:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508914304:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414454:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508914304:12345-0@lo:400:kworker.0 Request processed in 43us (1390us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.414461:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 886 00000100:00100000:2.0:1789760726.414471:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 887 00000100:00100000:2.0:1789760726.414476:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508914816:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414505:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508914816:12345-0@lo:400:kworker.0 Request processed in 29us (1371us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.414511:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 887 00000100:00100000:2.0:1789760726.414534:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 888 00000100:00100000:2.0:1789760726.414539:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508914048:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414569:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508914048:12345-0@lo:400:kworker.0 Request processed in 30us (550us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.414575:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 888 00000100:00100000:2.0:1789760726.414586:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 889 00000100:00100000:2.0:1789760726.414591:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3479:x1876699508914432:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414614:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3479:x1876699508914432:12345-0@lo:400:kworker.0 Request processed in 24us (441us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.414620:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 889 00000100:00100000:2.0:1789760726.414631:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 890 00000100:00100000:2.0:1789760726.414636:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3479:x1876699508914944:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414660:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3479:x1876699508914944:12345-0@lo:400:kworker.0 Request processed in 24us (448us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.414664:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 890 00000100:00100000:2.0:1789760726.414682:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508914560 00000100:00100000:2.0:1789760726.414686:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508914560 00000100:00100000:2.0:1789760726.414717:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 313 00000100:00100000:2.0:1789760726.414723:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508914560:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414753:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508914560:12345-0@lo:400:kworker.0 Request processed in 30us (687us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.414759:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 313 00000100:00100000:2.0:1789760726.414777:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508914176 00000100:00100000:2.0:1789760726.414780:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508914176 00000100:00100000:2.0:1789760726.414806:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508913920 00000100:00100000:2.0:1789760726.414809:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508913920 00000100:00100000:2.0:1789760726.414822:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 427 00000100:00100000:2.0:1789760726.414844:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508914176:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414878:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508914176:12345-0@lo:400:kworker.0 Request processed in 50us (1969us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.414884:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 427 00000100:00100000:2.0:1789760726.414896:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 428 00000100:00100000:2.0:1789760726.414901:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508913920:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414927:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508913920:12345-0@lo:400:kworker.0 Request processed in 26us (791us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760726.414932:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 428 00000100:00100000:2.0:1789760726.414976:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5081180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508913792:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414988:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5082a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508914304:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.414999:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5082d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508914816:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.415010:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5081880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508914176:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.415051:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5080000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508913920:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.415060:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5083480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0000_UUID:3479:1876699508914432:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.415095:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5081500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0001_UUID:3479:1876699508914944:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.415103:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5082300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508914048:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.415113:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5083800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508914560:0@lo:400:kworker.0 00000100:00100000:2.0:1789760726.415153:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c10700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508913536:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760726.415170:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c12a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508913408:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760726.415676:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508913664 00000100:00100000:1.0:1789760726.415685:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508913664 00000100:00100000:1.0:1789760726.415725:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 472 00000100:00100000:1.0:1789760726.415732:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b5450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3480:x1876699508913664:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760726.415791:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b5450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3480:x1876699508913664:12345-0@lo:400:kworker.0 Request processed in 59us (2936us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760726.415809:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 472 00000100:00080000:3.0:1789760726.416101:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:3.0:1789760726.416108:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:3.0:1789760726.416110:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:3.0:1789760726.416113:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1789760726.416302:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5083100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3480:1876699508913664:0@lo:400:kworker.0 00000001:00080000:0.0:1789760726.423490:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972219 is committed 00002000:00010000:2.0:1789760726.423691:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20074496 -> 20078592 00010000:00010000:2.0:1789760726.423712:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20074496->20078591] (req 20074496->20078591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.423728:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20074496->20078591] (req 20074496->20078591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.423739:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20074496->20078591] (req 20074496->20078591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.423748:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760726.423754:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20074496->20078591] (req 20074496->20078591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.423766:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20074496->20078591] (req 20074496->20078591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.423792:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20074496->20078591] (req 20074496->20078591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.423800:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396207fc0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20074496->20078591] (req 20074496->20078591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760726.423920:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484190208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58501us (58798us total) trans 4294972219 rc 0/0 00000100:00100000:2.0:1789760726.423932:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4930 00000100:00100000:2.0:1789760726.423944:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715627818546 time 715696331212 state 0x2 ref 0 00000100:00100000:0.0:1789760726.427265:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760726.427373:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484190336 00000100:00100000:1.0:1789760726.427459:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484190336 00000100:00100000:1.0:1789760726.427534:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715627818546 time 715696331212 state 0x0 ref 1 00000100:00100000:1.0:1789760726.427551:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715696591628 time 715696331212 state 0x1 ref 1 00000100:00100000:1.0:1789760726.427626:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715696591628 time 715758555949 state 0x0 ref 1 00000100:00100000:1.0:1789760726.427641:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484190336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760726.427707:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20078592, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:1.0:1789760726.427718:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.427737:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:1.0:1789760726.427742:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760726.427749:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20078592->20082687] (req 20078592->20082687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.427780:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20078592->20082687] (req 20078592->20082687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760726.481746:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972220 is committed 00002000:00010000:2.0:1789760726.482398:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20078592 -> 20082688 00010000:00010000:2.0:1789760726.482420:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20078592->20082687] (req 20078592->20082687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.482443:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20078592->20082687] (req 20078592->20082687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.482453:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20078592->20082687] (req 20078592->20082687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.482460:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760726.482465:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20078592->20082687] (req 20078592->20082687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.482475:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20078592->20082687] (req 20078592->20082687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.482504:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20078592->20082687] (req 20078592->20082687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.482512:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396207fc7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20078592->20082687] (req 20078592->20082687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760726.482641:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484190336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54995us (55384us total) trans 4294972220 rc 0/0 00000100:00100000:2.0:1789760726.482654:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4931 00000100:00100000:2.0:1789760726.482667:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715696591628 time 715758555949 state 0x2 ref 0 00000100:00100000:1.0:1789760726.486141:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760726.486370:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484190464 00000100:00100000:3.0:1789760726.486393:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484190464 00000100:00100000:3.0:1789760726.486473:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715696591628 time 715758555949 state 0x0 ref 1 00000100:00100000:3.0:1789760726.486497:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715758816365 time 715758555949 state 0x1 ref 1 00000100:00100000:3.0:1789760726.486529:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715758816365 time 715817459571 state 0x0 ref 1 00000100:00100000:3.0:1789760726.486544:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484190464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760726.486609:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20082688, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:3.0:1789760726.486619:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.486639:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:3.0:1789760726.486645:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760726.486653:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20082688->20086783] (req 20082688->20086783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.486675:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20082688->20086783] (req 20082688->20086783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760726.512352:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972221 is committed 00002000:00010000:1.0:1789760726.512570:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20082688 -> 20086784 00010000:00010000:1.0:1789760726.512601:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20082688->20086783] (req 20082688->20086783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.512619:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20082688->20086783] (req 20082688->20086783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.512630:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20082688->20086783] (req 20082688->20086783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.512639:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760726.512645:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20082688->20086783] (req 20082688->20086783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.512657:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20082688->20086783] (req 20082688->20086783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.512689:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20082688->20086783] (req 20082688->20086783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.512698:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396207fce lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20082688->20086783] (req 20082688->20086783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760726.512849:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484190464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26297us (26714us total) trans 4294972221 rc 0/0 00000100:00100000:1.0:1789760726.512863:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4932 00000100:00100000:1.0:1789760726.512880:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715758816365 time 715817459571 state 0x2 ref 0 00000100:00100000:3.0:1789760726.518364:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760726.518696:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484190592 00000100:00100000:0.0:1789760726.518712:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484190592 00000100:00100000:0.0:1789760726.518799:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715758816365 time 715817459571 state 0x0 ref 1 00000100:00100000:0.0:1789760726.518820:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715817719987 time 715817459571 state 0x1 ref 1 00000100:00100000:0.0:1789760726.518857:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715817719987 time 715849787896 state 0x0 ref 1 00000100:00100000:0.0:1789760726.518874:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484190592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760726.518947:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20086784, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:0.0:1789760726.518959:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.518980:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:0.0:1789760726.518985:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760726.518993:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20086784->20090879] (req 20086784->20090879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.519010:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20086784->20090879] (req 20086784->20090879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760726.568796:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20086784 -> 20090880 00010000:00010000:1.0:1789760726.568826:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20086784->20090879] (req 20086784->20090879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.568843:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20086784->20090879] (req 20086784->20090879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.568853:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20086784->20090879] (req 20086784->20090879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.568862:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760726.568868:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20086784->20090879] (req 20086784->20090879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.568879:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20086784->20090879] (req 20086784->20090879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.568905:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20086784->20090879] (req 20086784->20090879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.568919:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396207fd5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20086784->20090879] (req 20086784->20090879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760726.569056:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484190592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50178us (50697us total) trans 4294972222 rc 0/0 00000100:00100000:1.0:1789760726.569685:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4933 00000100:00100000:1.0:1789760726.569697:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715817719987 time 715849787896 state 0x2 ref 0 00000001:00080000:0.0:1789760726.569847:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972222 is committed 00000100:00100000:0.0:1789760726.571664:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760726.571762:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484190720 00000100:00100000:2.0:1789760726.571776:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484190720 00000100:00100000:2.0:1789760726.571849:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715817719987 time 715849787896 state 0x0 ref 1 00000100:00100000:2.0:1789760726.571868:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715850048312 time 715849787896 state 0x1 ref 1 00000100:00100000:2.0:1789760726.571937:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715850048312 time 715902867418 state 0x0 ref 1 00000100:00100000:2.0:1789760726.571952:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484190720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760726.572015:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20090880, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:2.0:1789760726.572027:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.572046:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:2.0:1789760726.572052:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760726.572103:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20090880->20094975] (req 20090880->20094975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.572122:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20090880->20094975] (req 20090880->20094975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760726.621953:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972223 is committed 00002000:00010000:1.0:1789760726.621980:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20090880 -> 20094976 00010000:00010000:1.0:1789760726.622001:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20090880->20094975] (req 20090880->20094975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.622017:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20090880->20094975] (req 20090880->20094975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.622028:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20090880->20094975] (req 20090880->20094975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.622036:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760726.622041:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20090880->20094975] (req 20090880->20094975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.622052:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20090880->20094975] (req 20090880->20094975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.622128:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20090880->20094975] (req 20090880->20094975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.622137:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396207fdc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20090880->20094975] (req 20090880->20094975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760726.622259:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484190720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50303us (50602us total) trans 4294972223 rc 0/0 00000100:00100000:1.0:1789760726.622272:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4934 00000100:00100000:1.0:1789760726.622284:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715850048312 time 715902867418 state 0x2 ref 0 00000100:00100000:2.0:1789760726.629600:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760726.629812:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484190848 00000100:00100000:1.0:1789760726.629826:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484190848 00000100:00100000:1.0:1789760726.629936:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715850048312 time 715902867418 state 0x0 ref 1 00000100:00100000:1.0:1789760726.629955:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715903127834 time 715902867418 state 0x1 ref 1 00000100:00100000:0.0:1789760726.630191:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715903127834 time 715961117446 state 0x0 ref 1 00000100:00100000:0.0:1789760726.630218:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484190848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760726.630284:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20094976, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:0.0:1789760726.630306:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.630328:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:0.0:1789760726.630333:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760726.630342:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20094976->20099071] (req 20094976->20099071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.630360:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20094976->20099071] (req 20094976->20099071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760726.657699:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972224 is committed 00002000:00010000:1.0:1789760726.657787:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20094976 -> 20099072 00010000:00010000:1.0:1789760726.657806:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20094976->20099071] (req 20094976->20099071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.657819:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20094976->20099071] (req 20094976->20099071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.657827:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20094976->20099071] (req 20094976->20099071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.657837:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760726.657841:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20094976->20099071] (req 20094976->20099071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.657849:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20094976->20099071] (req 20094976->20099071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.657868:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20094976->20099071] (req 20094976->20099071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.657873:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396207fe3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20094976->20099071] (req 20094976->20099071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760726.658018:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484190848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27806us (28425us total) trans 4294972224 rc 0/0 00000100:00100000:1.0:1789760726.658030:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4935 00000100:00100000:1.0:1789760726.658042:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715903127834 time 715961117446 state 0x2 ref 0 00000100:00100000:0.0:1789760726.662395:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760726.662511:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484190976 00000100:00100000:1.0:1789760726.662582:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484190976 00000100:00100000:1.0:1789760726.662661:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715903127834 time 715961117446 state 0x0 ref 1 00000100:00100000:1.0:1789760726.662681:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715961377862 time 715961117446 state 0x1 ref 1 00000100:00100000:1.0:1789760726.662759:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715961377862 time 715993688975 state 0x0 ref 1 00000100:00100000:1.0:1789760726.662779:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484190976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760726.662844:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20099072, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:1.0:1789760726.662855:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.662875:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=715s v=(0 0) 00010000:00010000:1.0:1789760726.662880:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760726.662888:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20099072->20103167] (req 20099072->20103167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.662906:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20099072->20103167] (req 20099072->20103167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760726.698098:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972225 is committed 00002000:00010000:3.0:1789760726.698371:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20099072 -> 20103168 00010000:00010000:3.0:1789760726.698392:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20099072->20103167] (req 20099072->20103167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.698409:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20099072->20103167] (req 20099072->20103167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.698419:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20099072->20103167] (req 20099072->20103167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.698437:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760726.698442:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20099072->20103167] (req 20099072->20103167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.698453:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20099072->20103167] (req 20099072->20103167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.698479:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20099072->20103167] (req 20099072->20103167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.698594:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396207fea lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20099072->20103167] (req 20099072->20103167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760726.698734:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484190976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35951us (36348us total) trans 4294972225 rc 0/0 00000100:00100000:3.0:1789760726.698746:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4936 00000100:00100000:3.0:1789760726.701750:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715961377862 time 715993688975 state 0x2 ref 0 00000100:00100000:0.0:1789760726.702273:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760726.702402:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484191104 00000100:00100000:2.0:1789760726.702414:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484191104 00000100:00100000:2.0:1789760726.702498:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715961377862 time 715993688975 state 0x0 ref 1 00000100:00100000:2.0:1789760726.702515:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715993949391 time 715993688975 state 0x1 ref 1 00000100:00100000:2.0:1789760726.702553:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715993949391 time 716033483739 state 0x0 ref 1 00000100:00100000:2.0:1789760726.702569:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484191104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760726.702637:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20103168, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:2.0:1789760726.702649:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.702667:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:2.0:1789760726.702672:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760726.702679:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20103168->20107263] (req 20103168->20107263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.702706:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20103168->20107263] (req 20103168->20107263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760726.745310:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972226 is committed 00002000:00010000:1.0:1789760726.745885:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20103168 -> 20107264 00010000:00010000:1.0:1789760726.745904:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20103168->20107263] (req 20103168->20107263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.745920:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20103168->20107263] (req 20103168->20107263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.745931:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20103168->20107263] (req 20103168->20107263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.745939:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760726.745944:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20103168->20107263] (req 20103168->20107263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.745956:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20103168->20107263] (req 20103168->20107263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.745979:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20103168->20107263] (req 20103168->20107263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.745987:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396207ff1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20103168->20107263] (req 20103168->20107263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760726.746149:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484191104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43576us (43875us total) trans 4294972226 rc 0/0 00000100:00100000:1.0:1789760726.746162:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4937 00000100:00100000:1.0:1789760726.746175:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715993949391 time 716033483739 state 0x2 ref 0 00000100:00100000:2.0:1789760726.758638:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760726.759360:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484191232 00000100:00100000:1.0:1789760726.759375:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484191232 00000100:00100000:1.0:1789760726.759479:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 715993949391 time 716033483739 state 0x0 ref 1 00000100:00100000:1.0:1789760726.759501:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716033744155 time 716033483739 state 0x1 ref 1 00000100:00100000:1.0:1789760726.759541:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716033744155 time 716090471646 state 0x0 ref 1 00000100:00100000:1.0:1789760726.759558:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484191232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760726.759631:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20107264, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:1.0:1789760726.759643:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.759665:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:1.0:1789760726.759670:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760726.759679:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20107264->20111359] (req 20107264->20111359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.759695:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20107264->20111359] (req 20107264->20111359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760726.813983:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972227 is committed 00002000:00010000:3.0:1789760726.814303:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20107264 -> 20111360 00010000:00010000:3.0:1789760726.814325:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20107264->20111359] (req 20107264->20111359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.814340:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20107264->20111359] (req 20107264->20111359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.814350:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20107264->20111359] (req 20107264->20111359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.814357:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760726.814362:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20107264->20111359] (req 20107264->20111359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.814372:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20107264->20111359] (req 20107264->20111359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.814396:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20107264->20111359] (req 20107264->20111359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.814408:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396207ff8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20107264->20111359] (req 20107264->20111359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760726.814545:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484191232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54984us (55916us total) trans 4294972227 rc 0/0 00000100:00100000:3.0:1789760726.814558:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4938 00000100:00100000:3.0:1789760726.814572:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716033744155 time 716090471646 state 0x2 ref 0 00000100:00100000:0.0:1789760726.830491:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760726.830695:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484191360 00000100:00100000:1.0:1789760726.830709:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484191360 00000100:00100000:0.0:1789760726.830874:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716033744155 time 716090471646 state 0x0 ref 1 00000100:00100000:0.0:1789760726.830894:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716090732062 time 716090471646 state 0x1 ref 1 00000100:00100000:0.0:1789760726.830973:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716090732062 time 716161902961 state 0x0 ref 1 00000100:00100000:0.0:1789760726.830990:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484191360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760726.831112:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20111360, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:0.0:1789760726.831122:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.831147:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:0.0:1789760726.831153:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760726.831161:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20111360->20115455] (req 20111360->20115455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760726.831180:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20111360->20115455] (req 20111360->20115455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760726.895138:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972228 is committed 00002000:00010000:2.0:1789760726.895657:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20111360 -> 20115456 00010000:00010000:2.0:1789760726.895678:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20111360->20115455] (req 20111360->20115455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.895694:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20111360->20115455] (req 20111360->20115455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.895705:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20111360->20115455] (req 20111360->20115455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.895713:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760726.895719:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20111360->20115455] (req 20111360->20115455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.895730:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20111360->20115455] (req 20111360->20115455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.895756:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20111360->20115455] (req 20111360->20115455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.895770:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396207fff lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20111360->20115455] (req 20111360->20115455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760726.895900:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484191360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 64907us (65417us total) trans 4294972228 rc 0/0 00000100:00100000:2.0:1789760726.895913:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4939 00000100:00100000:2.0:1789760726.895925:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716090732062 time 716161902961 state 0x2 ref 0 00000100:00100000:0.0:1789760726.906507:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760726.906829:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484191488 00000100:00100000:3.0:1789760726.906841:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484191488 00000100:00100000:3.0:1789760726.906906:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716090732062 time 716161902961 state 0x0 ref 1 00000100:00100000:3.0:1789760726.906925:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716162163377 time 716161902961 state 0x1 ref 1 00000100:00100000:3.0:1789760726.906972:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716162163377 time 716237902424 state 0x0 ref 1 00000100:00100000:3.0:1789760726.906989:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484191488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760726.907058:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20115456, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:3.0:1789760726.907115:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.907138:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:3.0:1789760726.907143:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760726.907151:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20115456->20119551] (req 20115456->20119551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.907170:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20115456->20119551] (req 20115456->20119551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760726.925357:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:3.0:1789760726.939800:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20115456 -> 20119552 00010000:00010000:3.0:1789760726.939827:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20115456->20119551] (req 20115456->20119551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.939842:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20115456->20119551] (req 20115456->20119551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.939853:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20115456->20119551] (req 20115456->20119551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.939861:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760726.939867:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20115456->20119551] (req 20115456->20119551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.939877:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20115456->20119551] (req 20115456->20119551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.939901:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20115456->20119551] (req 20115456->20119551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760726.939909:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208006 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20115456->20119551] (req 20115456->20119551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760726.940038:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484191488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33045us (33539us total) trans 4294972229 rc 0/0 00000100:00100000:3.0:1789760726.940052:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4940 00000100:00100000:3.0:1789760726.940111:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716162163377 time 716237902424 state 0x2 ref 0 00000001:00080000:3.0:1789760726.940167:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972229 is committed 00000100:00100000:3.0:1789760726.945462:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760726.945872:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484191616 00000100:00100000:1.0:1789760726.945887:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484191616 00000100:00100000:1.0:1789760726.945977:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716162163377 time 716237902424 state 0x0 ref 1 00000100:00100000:1.0:1789760726.945997:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716238162840 time 716237902424 state 0x1 ref 1 00000100:00100000:1.0:1789760726.946124:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716238162840 time 716277053411 state 0x0 ref 1 00000100:00100000:1.0:1789760726.946140:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484191616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760726.946210:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20119552, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:1.0:1789760726.946220:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.946241:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:1.0:1789760726.946247:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760726.946255:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20119552->20123647] (req 20119552->20123647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760726.946273:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20119552->20123647] (req 20119552->20123647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760727.013489:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20119552 -> 20123648 00010000:00010000:1.0:1789760727.013520:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20119552->20123647] (req 20119552->20123647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.013536:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20119552->20123647] (req 20119552->20123647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.013545:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20119552->20123647] (req 20119552->20123647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.013553:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760727.013559:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20119552->20123647] (req 20119552->20123647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.013569:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20119552->20123647] (req 20119552->20123647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.013596:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20119552->20123647] (req 20119552->20123647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.013604:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620800d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20119552->20123647] (req 20119552->20123647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760727.013618:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972230 is committed 00000100:00100000:1.0:1789760727.013842:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484191616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67699us (68389us total) trans 4294972230 rc 0/0 00000100:00100000:1.0:1789760727.013862:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4941 00000100:00100000:1.0:1789760727.013875:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716238162840 time 716277053411 state 0x2 ref 0 00000100:00100000:0.0:1789760727.022146:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760727.023287:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484191744 00000100:00100000:1.0:1789760727.023306:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484191744 00000100:00100000:1.0:1789760727.023387:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716238162840 time 716277053411 state 0x0 ref 1 00000100:00100000:1.0:1789760727.023406:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716277313827 time 716277053411 state 0x1 ref 1 00000100:00100000:1.0:1789760727.023451:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716277313827 time 716354381961 state 0x0 ref 1 00000100:00100000:1.0:1789760727.023466:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484191744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760727.023543:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20123648, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:1.0:1789760727.023554:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.023574:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:1.0:1789760727.023579:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760727.023587:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20123648->20127743] (req 20123648->20127743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.023605:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20123648->20127743] (req 20123648->20127743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760727.102668:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972231 is committed 00002000:00010000:2.0:1789760727.102856:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20123648 -> 20127744 00010000:00010000:2.0:1789760727.102892:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20123648->20127743] (req 20123648->20127743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.102908:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20123648->20127743] (req 20123648->20127743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.102918:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20123648->20127743] (req 20123648->20127743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.102925:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760727.102940:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20123648->20127743] (req 20123648->20127743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.102952:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20123648->20127743] (req 20123648->20127743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.102984:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20123648->20127743] (req 20123648->20127743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.102993:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208014 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20123648->20127743] (req 20123648->20127743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760727.103174:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484191744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79704us (81038us total) trans 4294972231 rc 0/0 00000100:00100000:2.0:1789760727.103197:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4942 00000100:00100000:2.0:1789760727.103219:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716277313827 time 716354381961 state 0x2 ref 0 00000100:00100000:1.0:1789760727.110607:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760727.110774:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484191872 00000100:00100000:2.0:1789760727.110786:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484191872 00000100:00100000:2.0:1789760727.110866:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716277313827 time 716354381961 state 0x0 ref 1 00000100:00100000:2.0:1789760727.110894:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716354642377 time 716354381961 state 0x1 ref 1 00000100:00100000:2.0:1789760727.110926:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716354642377 time 716441856771 state 0x0 ref 1 00000100:00100000:2.0:1789760727.110940:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484191872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760727.111006:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20127744, mtime: 0x6aad94d6, atime: 0x0, ctime: 0x6aad94d6, blocks: 0xa007 00010000:00010000:2.0:1789760727.111017:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.111037:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:2.0:1789760727.111043:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760727.111050:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20127744->20131839] (req 20127744->20131839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.111093:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20127744->20131839] (req 20127744->20131839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760727.172938:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972232 is committed 00002000:00010000:2.0:1789760727.173420:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20127744 -> 20131840 00002000:00010000:2.0:1789760727.173434:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760726 -> 1789760727 00002000:00010000:2.0:1789760727.173437:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760726 -> 1789760727 00010000:00010000:2.0:1789760727.173455:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20127744->20131839] (req 20127744->20131839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.173472:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20127744->20131839] (req 20127744->20131839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.173482:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20127744->20131839] (req 20127744->20131839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.173491:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760727.173497:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20127744->20131839] (req 20127744->20131839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.173508:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20127744->20131839] (req 20127744->20131839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.173544:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20127744->20131839] (req 20127744->20131839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.173554:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620801b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20127744->20131839] (req 20127744->20131839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760727.173728:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484191872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62782us (63128us total) trans 4294972232 rc 0/0 00000100:00100000:2.0:1789760727.173741:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4943 00000100:00100000:2.0:1789760727.173754:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716354642377 time 716441856771 state 0x2 ref 0 00000100:00100000:1.0:1789760727.182869:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760727.183026:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484192000 00000100:00100000:3.0:1789760727.183040:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484192000 00000100:00100000:3.0:1789760727.183170:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716354642377 time 716441856771 state 0x0 ref 1 00000100:00100000:3.0:1789760727.183304:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716442117187 time 716441856771 state 0x1 ref 1 00000100:00100000:3.0:1789760727.183343:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716442117187 time 716514272861 state 0x0 ref 1 00000100:00100000:3.0:1789760727.183357:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484192000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760727.183432:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20131840, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:3.0:1789760727.183443:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.183465:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:3.0:1789760727.183470:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760727.183479:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20131840->20135935] (req 20131840->20135935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.183495:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20131840->20135935] (req 20131840->20135935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760727.232178:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972233 is committed 00002000:00010000:2.0:1789760727.235734:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20131840 -> 20135936 00010000:00010000:2.0:1789760727.235754:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20131840->20135935] (req 20131840->20135935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.235769:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20131840->20135935] (req 20131840->20135935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.235778:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20131840->20135935] (req 20131840->20135935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.235786:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760727.235790:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20131840->20135935] (req 20131840->20135935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.235801:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20131840->20135935] (req 20131840->20135935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.235826:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20131840->20135935] (req 20131840->20135935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.235834:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208022 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20131840->20135935] (req 20131840->20135935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760727.235999:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484192000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52640us (53139us total) trans 4294972233 rc 0/0 00000100:00100000:2.0:1789760727.236010:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4944 00000100:00100000:2.0:1789760727.236020:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716442117187 time 716514272861 state 0x2 ref 0 00000100:00100000:2.0:1789760727.243941:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760727.244124:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484192128 00000100:00100000:1.0:1789760727.244134:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484192128 00000100:00100000:1.0:1789760727.244187:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716442117187 time 716514272861 state 0x0 ref 1 00000100:00100000:1.0:1789760727.244203:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716514533277 time 716514272861 state 0x1 ref 1 00000100:00100000:1.0:1789760727.244317:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716514533277 time 716575247315 state 0x0 ref 1 00000100:00100000:1.0:1789760727.244332:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484192128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760727.244385:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20135936, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:1.0:1789760727.244394:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.244411:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:1.0:1789760727.244416:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760727.244433:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20135936->20140031] (req 20135936->20140031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.244458:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20135936->20140031] (req 20135936->20140031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760727.269516:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760727.269745:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484192256 00000100:00100000:0.0:1789760727.269755:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484192256 00000100:00100000:0.0:1789760727.269967:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 473 00000100:00100000:0.0:1789760727.269980:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b4450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2425:x1876699484192256:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760727.270053:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b4450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2425:x1876699484192256:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 76us (545us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760727.270060:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 473 00000100:00100000:3.0:1789760727.270467:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760727.270521:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484192384 00000100:00100000:1.0:1789760727.270530:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484192384 00000100:00100000:1.0:1789760727.270598:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 891 00000100:00100000:1.0:1789760727.270611:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484192384:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760727.270667:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2425:x1876699484192384:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 61us (210us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760727.270674:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 891 00000100:00080000:0.0:1789760727.270731:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:3.0:1789760727.276741:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760727.276793:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760727.276823:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760727.276860:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484192512 00000100:00100000:3.0:1789760727.276872:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484192512 00000100:00100000:1.0:1789760727.276929:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484192768 00000100:00100000:1.0:1789760727.276945:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484192768 00000100:00100000:3.0:1789760727.276954:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484192640 00000100:00100000:3.0:1789760727.276960:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484192640 00000100:00100000:3.0:1789760727.277004:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 892 00000100:00100000:3.0:1789760727.277017:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484192512:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760727.277030:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 893 00000100:00100000:1.0:1789760727.277044:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484192640:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760727.277147:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484192512:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 131us (419us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760727.277157:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484192640:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 111us (326us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760727.277165:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 892 00000100:00100000:1.0:1789760727.277178:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 893 00000100:00100000:3.0:1789760727.277202:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 894 00000100:00100000:3.0:1789760727.277752:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484192768:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760727.277803:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484192768:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 60us (1009us total) trans 0 rc 0/0 00000100:00080000:2.0:1789760727.280627:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:2.0:1789760727.280641:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:3.0:1789760727.280722:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 894 00000001:00080000:3.0:1789760727.310094:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972234 is committed 00002000:00010000:2.0:1789760727.310215:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20135936 -> 20140032 00010000:00010000:2.0:1789760727.310230:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20135936->20140031] (req 20135936->20140031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.310243:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20135936->20140031] (req 20135936->20140031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.310252:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20135936->20140031] (req 20135936->20140031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.310259:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760727.310264:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20135936->20140031] (req 20135936->20140031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.310273:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20135936->20140031] (req 20135936->20140031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.310293:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20135936->20140031] (req 20135936->20140031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.310301:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208029 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20135936->20140031] (req 20135936->20140031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760727.310395:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484192128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66062us (66458us total) trans 4294972234 rc 0/0 00000100:00100000:2.0:1789760727.310405:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4945 00000100:00100000:2.0:1789760727.310415:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716514533277 time 716575247315 state 0x2 ref 0 00000100:00100000:3.0:1789760727.314135:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760727.314314:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484192896 00000100:00100000:2.0:1789760727.314324:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484192896 00000100:00100000:2.0:1789760727.314378:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716514533277 time 716575247315 state 0x0 ref 1 00000100:00100000:2.0:1789760727.314392:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716575507731 time 716575247315 state 0x1 ref 1 00000100:00100000:2.0:1789760727.314432:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716575507731 time 716645363474 state 0x0 ref 1 00000100:00100000:2.0:1789760727.314445:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484192896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760727.314492:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20140032, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:2.0:1789760727.314502:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.314518:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:2.0:1789760727.314522:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760727.314529:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20140032->20144127] (req 20140032->20144127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.314543:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20140032->20144127] (req 20140032->20144127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760727.364600:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972235 is committed 00002000:00010000:2.0:1789760727.365026:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20140032 -> 20144128 00010000:00010000:2.0:1789760727.365044:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20140032->20144127] (req 20140032->20144127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.365057:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20140032->20144127] (req 20140032->20144127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.365107:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20140032->20144127] (req 20140032->20144127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.365115:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760727.365120:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20140032->20144127] (req 20140032->20144127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.365129:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20140032->20144127] (req 20140032->20144127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.365148:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20140032->20144127] (req 20140032->20144127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.365157:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208030 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20140032->20144127] (req 20140032->20144127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760727.365502:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484192896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51053us (51369us total) trans 4294972235 rc 0/0 00000100:00100000:2.0:1789760727.365514:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4946 00000100:00100000:2.0:1789760727.365523:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716575507731 time 716645363474 state 0x2 ref 0 00000100:00100000:3.0:1789760727.371502:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760727.371584:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484193024 00000100:00100000:2.0:1789760727.371595:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484193024 00000100:00100000:2.0:1789760727.371649:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716575507731 time 716645363474 state 0x0 ref 1 00000100:00100000:2.0:1789760727.371665:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716645623890 time 716645363474 state 0x1 ref 1 00000100:00100000:2.0:1789760727.371700:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716645623890 time 716702630842 state 0x0 ref 1 00000100:00100000:2.0:1789760727.371714:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484193024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760727.371761:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20144128, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:2.0:1789760727.371770:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.371786:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:2.0:1789760727.371791:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760727.371806:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20144128->20148223] (req 20144128->20148223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.371826:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20144128->20148223] (req 20144128->20148223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760727.409853:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972236 is committed 00002000:00010000:2.0:1789760727.409965:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20144128 -> 20148224 00010000:00010000:2.0:1789760727.410051:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20144128->20148223] (req 20144128->20148223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.410084:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20144128->20148223] (req 20144128->20148223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.410094:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20144128->20148223] (req 20144128->20148223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.410102:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760727.410107:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20144128->20148223] (req 20144128->20148223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.410116:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20144128->20148223] (req 20144128->20148223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.410134:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20144128->20148223] (req 20144128->20148223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.410141:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208037 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20144128->20148223] (req 20144128->20148223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760727.410558:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484193024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38841us (39059us total) trans 4294972236 rc 0/0 00000100:00100000:2.0:1789760727.410569:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4947 00000100:00100000:2.0:1789760727.410579:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716645623890 time 716702630842 state 0x2 ref 0 00000100:00100000:3.0:1789760727.416101:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760727.416186:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484193152 00000100:00100000:0.0:1789760727.416196:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484193152 00000100:00100000:0.0:1789760727.416269:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716645623890 time 716702630842 state 0x0 ref 1 00000100:00100000:0.0:1789760727.416286:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716702891258 time 716702630842 state 0x1 ref 1 00000100:00100000:0.0:1789760727.416316:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716702891258 time 716747247554 state 0x0 ref 1 00000100:00100000:0.0:1789760727.416330:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484193152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760727.416378:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20148224, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:0.0:1789760727.416387:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.416402:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:0.0:1789760727.416406:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760727.416412:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20148224->20152319] (req 20148224->20152319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.416583:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20148224->20152319] (req 20148224->20152319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760727.465979:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972237 is committed 00002000:00010000:2.0:1789760727.466218:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20148224 -> 20152320 00010000:00010000:2.0:1789760727.466234:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20148224->20152319] (req 20148224->20152319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.466247:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20148224->20152319] (req 20148224->20152319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.466256:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20148224->20152319] (req 20148224->20152319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.466264:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760727.466269:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20148224->20152319] (req 20148224->20152319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.466278:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20148224->20152319] (req 20148224->20152319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.466297:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20148224->20152319] (req 20148224->20152319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.466312:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620803e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20148224->20152319] (req 20148224->20152319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760727.466403:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484193152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50071us (50341us total) trans 4294972237 rc 0/0 00000100:00100000:2.0:1789760727.466414:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4948 00000100:00100000:2.0:1789760727.466433:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716702891258 time 716747247554 state 0x2 ref 0 00000100:00100000:3.0:1789760727.469028:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760727.469156:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484193280 00000100:00100000:1.0:1789760727.469167:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484193280 00000100:00100000:1.0:1789760727.469232:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716702891258 time 716747247554 state 0x0 ref 1 00000100:00100000:1.0:1789760727.469248:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716747507970 time 716747247554 state 0x1 ref 1 00000100:00100000:1.0:1789760727.469276:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716747507970 time 716800207518 state 0x0 ref 1 00000100:00100000:1.0:1789760727.469291:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484193280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760727.469341:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20152320, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:1.0:1789760727.469350:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.469371:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:1.0:1789760727.469375:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760727.469382:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20152320->20156415] (req 20152320->20156415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.469395:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20152320->20156415] (req 20152320->20156415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760727.501633:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972238 is committed 00002000:00010000:2.0:1789760727.501999:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20152320 -> 20156416 00010000:00010000:2.0:1789760727.502013:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20152320->20156415] (req 20152320->20156415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.502027:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20152320->20156415] (req 20152320->20156415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.502037:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20152320->20156415] (req 20152320->20156415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.502043:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760727.502048:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20152320->20156415] (req 20152320->20156415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.502057:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20152320->20156415] (req 20152320->20156415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.502099:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20152320->20156415] (req 20152320->20156415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.502108:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208045 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20152320->20156415] (req 20152320->20156415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760727.502197:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484193280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32904us (33174us total) trans 4294972238 rc 0/0 00000100:00100000:2.0:1789760727.502268:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4949 00000100:00100000:2.0:1789760727.502277:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716747507970 time 716800207518 state 0x2 ref 0 00000100:00100000:3.0:1789760727.506124:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760727.506387:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484193408 00000100:00100000:2.0:1789760727.506396:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484193408 00000100:00100000:2.0:1789760727.506457:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716747507970 time 716800207518 state 0x0 ref 1 00000100:00100000:2.0:1789760727.506471:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716800467934 time 716800207518 state 0x1 ref 1 00000100:00100000:2.0:1789760727.506493:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716800467934 time 716837424282 state 0x0 ref 1 00000100:00100000:2.0:1789760727.506504:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484193408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760727.506562:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20156416, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:2.0:1789760727.506571:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.506587:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:2.0:1789760727.506592:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760727.506598:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20156416->20160511] (req 20156416->20160511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.506612:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20156416->20160511] (req 20156416->20160511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760727.534011:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20156416 -> 20160512 00010000:00010000:3.0:1789760727.534032:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20156416->20160511] (req 20156416->20160511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.534046:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20156416->20160511] (req 20156416->20160511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.534055:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20156416->20160511] (req 20156416->20160511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.534063:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760727.534096:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20156416->20160511] (req 20156416->20160511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.534106:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20156416->20160511] (req 20156416->20160511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.534125:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20156416->20160511] (req 20156416->20160511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.534132:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620804c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20156416->20160511] (req 20156416->20160511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760727.534229:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484193408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27721us (28107us total) trans 4294972239 rc 0/0 00000100:00100000:3.0:1789760727.534238:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4950 00000100:00100000:3.0:1789760727.534247:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716800467934 time 716837424282 state 0x2 ref 0 00000001:00080000:1.0:1789760727.535590:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972239 is committed 00000100:00100000:0.0:1789760727.538173:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760727.538234:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484193536 00000100:00100000:2.0:1789760727.538244:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484193536 00000100:00100000:2.0:1789760727.538299:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716800467934 time 716837424282 state 0x0 ref 1 00000100:00100000:2.0:1789760727.538314:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716837684698 time 716837424282 state 0x1 ref 1 00000100:00100000:2.0:1789760727.538366:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716837684698 time 716869298112 state 0x0 ref 1 00000100:00100000:2.0:1789760727.538376:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484193536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760727.538433:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20160512, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:2.0:1789760727.538442:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.538457:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:2.0:1789760727.538462:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760727.538467:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20160512->20164607] (req 20160512->20164607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.538481:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20160512->20164607] (req 20160512->20164607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760727.583197:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972240 is committed 00002000:00010000:1.0:1789760727.583895:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20160512 -> 20164608 00010000:00010000:1.0:1789760727.583911:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20160512->20164607] (req 20160512->20164607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.583924:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20160512->20164607] (req 20160512->20164607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.583933:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20160512->20164607] (req 20160512->20164607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.583941:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760727.583946:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20160512->20164607] (req 20160512->20164607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.583955:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20160512->20164607] (req 20160512->20164607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.583980:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20160512->20164607] (req 20160512->20164607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.583988:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208053 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20160512->20164607] (req 20160512->20164607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760727.584155:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484193536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45775us (46094us total) trans 4294972240 rc 0/0 00000100:00100000:1.0:1789760727.584166:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4951 00000100:00100000:1.0:1789760727.584176:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716837684698 time 716869298112 state 0x2 ref 0 00000100:00100000:0.0:1789760727.587647:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760727.588135:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484193664 00000100:00100000:2.0:1789760727.588146:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484193664 00000100:00100000:2.0:1789760727.588202:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716837684698 time 716869298112 state 0x0 ref 1 00000100:00100000:2.0:1789760727.588218:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716869558528 time 716869298112 state 0x1 ref 1 00000100:00100000:2.0:1789760727.588252:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716869558528 time 716919182768 state 0x0 ref 1 00000100:00100000:2.0:1789760727.588265:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484193664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760727.588314:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20164608, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:2.0:1789760727.588322:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.588338:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:2.0:1789760727.588343:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760727.588349:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20164608->20168703] (req 20164608->20168703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760727.588363:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20164608->20168703] (req 20164608->20168703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760727.635964:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972241 is committed 00002000:00010000:1.0:1789760727.636320:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20164608 -> 20168704 00010000:00010000:1.0:1789760727.636336:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20164608->20168703] (req 20164608->20168703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.636348:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20164608->20168703] (req 20164608->20168703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.636357:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20164608->20168703] (req 20164608->20168703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.636363:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760727.636367:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20164608->20168703] (req 20164608->20168703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.636383:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20164608->20168703] (req 20164608->20168703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.636403:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20164608->20168703] (req 20164608->20168703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.636411:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620805a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20164608->20168703] (req 20164608->20168703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760727.636577:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484193664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48309us (48934us total) trans 4294972241 rc 0/0 00000100:00100000:1.0:1789760727.636587:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4952 00000100:00100000:1.0:1789760727.636596:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716869558528 time 716919182768 state 0x2 ref 0 00000100:00100000:3.0:1789760727.647202:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760727.647276:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484193792 00000100:00100000:1.0:1789760727.647287:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484193792 00000100:00100000:1.0:1789760727.647343:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716869558528 time 716919182768 state 0x0 ref 1 00000100:00100000:1.0:1789760727.647359:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716919443184 time 716919182768 state 0x1 ref 1 00000100:00100000:1.0:1789760727.647395:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716919443184 time 716978326066 state 0x0 ref 1 00000100:00100000:1.0:1789760727.647409:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484193792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760727.647466:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20168704, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:1.0:1789760727.647538:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.647556:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=716s v=(0 0) 00010000:00010000:1.0:1789760727.647560:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760727.647567:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20168704->20172799] (req 20168704->20172799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.647581:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20168704->20172799] (req 20168704->20172799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760727.714332:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972242 is committed 00002000:00010000:1.0:1789760727.714445:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20168704 -> 20172800 00010000:00010000:1.0:1789760727.714459:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20168704->20172799] (req 20168704->20172799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.714477:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20168704->20172799] (req 20168704->20172799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.714487:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20168704->20172799] (req 20168704->20172799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.714494:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760727.714498:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20168704->20172799] (req 20168704->20172799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.714507:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20168704->20172799] (req 20168704->20172799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.714529:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20168704->20172799] (req 20168704->20172799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.714539:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208061 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20168704->20172799] (req 20168704->20172799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760727.714665:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484193792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67254us (67468us total) trans 4294972242 rc 0/0 00000100:00100000:1.0:1789760727.714676:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4953 00000100:00100000:1.0:1789760727.714685:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716919443184 time 716978326066 state 0x2 ref 0 00000100:00100000:1.0:1789760727.719532:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760727.719608:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484193920 00000100:00100000:3.0:1789760727.719618:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484193920 00000100:00100000:3.0:1789760727.719669:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716919443184 time 716978326066 state 0x0 ref 1 00000100:00100000:3.0:1789760727.719683:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716978586482 time 716978326066 state 0x1 ref 1 00000100:00100000:3.0:1789760727.719712:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716978586482 time 717050642325 state 0x0 ref 1 00000100:00100000:3.0:1789760727.719724:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484193920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760727.719771:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20172800, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:3.0:1789760727.719780:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.719794:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760727.719798:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760727.719806:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20172800->20176895] (req 20172800->20176895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.719826:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20172800->20176895] (req 20172800->20176895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760727.756768:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972243 is committed 00002000:00010000:1.0:1789760727.756882:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20172800 -> 20176896 00010000:00010000:1.0:1789760727.756896:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20172800->20176895] (req 20172800->20176895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.756907:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20172800->20176895] (req 20172800->20176895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.756916:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20172800->20176895] (req 20172800->20176895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.756923:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760727.756927:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20172800->20176895] (req 20172800->20176895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.756935:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20172800->20176895] (req 20172800->20176895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.756959:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20172800->20176895] (req 20172800->20176895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.756967:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208068 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20172800->20176895] (req 20172800->20176895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760727.757105:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484193920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37379us (37579us total) trans 4294972243 rc 0/0 00000100:00100000:1.0:1789760727.757116:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4954 00000100:00100000:1.0:1789760727.757125:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716978586482 time 717050642325 state 0x2 ref 0 00000100:00100000:0.0:1789760727.761668:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760727.761826:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484194048 00000100:00100000:2.0:1789760727.761836:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484194048 00000100:00100000:2.0:1789760727.761891:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 716978586482 time 717050642325 state 0x0 ref 1 00000100:00100000:2.0:1789760727.761905:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717050902741 time 717050642325 state 0x1 ref 1 00000100:00100000:3.0:1789760727.761971:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717050902741 time 717092899841 state 0x0 ref 1 00000100:00100000:3.0:1789760727.761987:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484194048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760727.762044:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20176896, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:3.0:1789760727.762053:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.762100:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760727.762106:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760727.762112:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20176896->20180991] (req 20176896->20180991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.762127:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20176896->20180991] (req 20176896->20180991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760727.807902:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972244 is committed 00002000:00010000:3.0:1789760727.808100:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20176896 -> 20180992 00010000:00010000:3.0:1789760727.808117:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20176896->20180991] (req 20176896->20180991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.808129:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20176896->20180991] (req 20176896->20180991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.808144:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20176896->20180991] (req 20176896->20180991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.808152:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760727.808157:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20176896->20180991] (req 20176896->20180991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.808165:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20176896->20180991] (req 20176896->20180991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.808183:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20176896->20180991] (req 20176896->20180991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.808192:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620806f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20176896->20180991] (req 20176896->20180991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760727.808322:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484194048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46334us (46660us total) trans 4294972244 rc 0/0 00000100:00100000:3.0:1789760727.808332:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4955 00000100:00100000:3.0:1789760727.808341:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717050902741 time 717092899841 state 0x2 ref 0 00000100:00100000:2.0:1789760727.814539:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760727.814909:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484194176 00000100:00100000:1.0:1789760727.814918:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484194176 00000100:00100000:1.0:1789760727.814966:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717050902741 time 717092899841 state 0x0 ref 1 00000100:00100000:1.0:1789760727.814979:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717093160257 time 717092899841 state 0x1 ref 1 00000100:00100000:1.0:1789760727.815010:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717093160257 time 717145941340 state 0x0 ref 1 00000100:00100000:1.0:1789760727.815021:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484194176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760727.815083:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20180992, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:1.0:1789760727.815092:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.815106:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:1.0:1789760727.815119:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760727.815125:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20180992->20185087] (req 20180992->20185087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.815138:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20180992->20185087] (req 20180992->20185087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760727.866928:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972245 is committed 00002000:00010000:1.0:1789760727.867034:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20180992 -> 20185088 00010000:00010000:1.0:1789760727.867050:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20180992->20185087] (req 20180992->20185087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.867062:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20180992->20185087] (req 20180992->20185087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.867094:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20180992->20185087] (req 20180992->20185087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.867102:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760727.867107:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20180992->20185087] (req 20180992->20185087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.867115:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20180992->20185087] (req 20180992->20185087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.867135:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20180992->20185087] (req 20180992->20185087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760727.867142:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208076 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20180992->20185087] (req 20180992->20185087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760727.867280:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484194176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52255us (52745us total) trans 4294972245 rc 0/0 00000100:00100000:1.0:1789760727.867290:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4956 00000100:00100000:1.0:1789760727.867300:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717093160257 time 717145941340 state 0x2 ref 0 00000100:00100000:1.0:1789760727.872395:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760727.872457:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484194304 00000100:00100000:3.0:1789760727.872467:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484194304 00000100:00100000:3.0:1789760727.872586:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717093160257 time 717145941340 state 0x0 ref 1 00000100:00100000:3.0:1789760727.872601:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717146201756 time 717145941340 state 0x1 ref 1 00000100:00100000:3.0:1789760727.872638:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717146201756 time 717203569019 state 0x0 ref 1 00000100:00100000:3.0:1789760727.872652:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484194304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760727.872701:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20185088, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:3.0:1789760727.872711:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.872730:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760727.872735:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760727.872742:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20185088->20189183] (req 20185088->20189183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.872755:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20185088->20189183] (req 20185088->20189183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760727.909627:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20185088 -> 20189184 00010000:00010000:0.0:1789760727.909647:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20185088->20189183] (req 20185088->20189183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.909660:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20185088->20189183] (req 20185088->20189183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.909669:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20185088->20189183] (req 20185088->20189183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.909682:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760727.909687:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20185088->20189183] (req 20185088->20189183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.909696:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20185088->20189183] (req 20185088->20189183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.909715:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20185088->20189183] (req 20185088->20189183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.909723:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620807d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20185088->20189183] (req 20185088->20189183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760727.909820:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484194304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37165us (37427us total) trans 4294972246 rc 0/0 00000100:00100000:0.0:1789760727.909829:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4957 00000100:00100000:0.0:1789760727.909838:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717146201756 time 717203569019 state 0x2 ref 0 00000001:00080000:2.0:1789760727.909923:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972246 is committed 00000100:00100000:2.0:1789760727.924052:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760727.924267:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484194432 00000100:00100000:3.0:1789760727.924278:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484194432 00000100:00100000:3.0:1789760727.924332:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717146201756 time 717203569019 state 0x0 ref 1 00000100:00100000:3.0:1789760727.924349:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717203829435 time 717203569019 state 0x1 ref 1 00000100:00100000:3.0:1789760727.924391:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717203829435 time 717255321958 state 0x0 ref 1 00000100:00100000:3.0:1789760727.924404:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484194432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760727.924472:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20189184, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:3.0:1789760727.924483:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.924498:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760727.924503:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760727.924509:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20189184->20193279] (req 20189184->20193279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.924523:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20189184->20193279] (req 20189184->20193279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760727.948236:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508915072:0@lo:41:osp-pre-0-1.0 00000100:00100000:1.0:1789760727.948298:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760727.948375:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508915072 00000100:00100000:1.0:1789760727.948381:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508915072 00000100:00100000:1.0:1789760727.948430:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 429 00000100:00100000:1.0:1789760727.948438:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508915072:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:1.0:1789760727.948471:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:1.0:1789760727.948508:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ace00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508915072:12345-0@lo:41:osp-pre-0-1.0 Request processed in 71us (209us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760727.948515:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 429 00000100:00100000:3.0:1789760727.950676:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5083b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508915200:0@lo:13:osp-pre-0-1.0 00000100:00100000:3.0:1789760727.950737:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760727.950782:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1789760727.950789:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c880e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508915328:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760727.950815:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00010000:00010000:3.0:1789760727.950846:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760727.950919:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508915200 00000100:00100000:1.0:1789760727.952141:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508915072:0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760727.952274:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508915200 00000100:00100000:3.0:1789760727.952297:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508915328 00000100:00100000:3.0:1789760727.952302:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508915328 00000100:00100000:3.0:1789760727.952342:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 370 00000100:00100000:3.0:1789760727.952351:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508915328:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:3.0:1789760727.952372:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528416768 pending 1703936 free 3837788160 avail 3831496704 00000100:00100000:3.0:1789760727.952558:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508915328:12345-0@lo:13:osp-pre-0-0.0 Request processed in 207us (1740us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760727.952566:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 370 00000100:00100000:3.0:1789760727.952608:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c880e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508915328:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760727.952651:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 371 00000100:00100000:3.0:1789760727.952658:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508915200:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:3.0:1789760727.952668:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528416768 pending 1703936 free 3837788160 avail 3831496704 00000100:00100000:3.0:1789760727.952699:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508915200:12345-0@lo:13:osp-pre-0-1.0 Request processed in 42us (1963us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760727.952705:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 371 00000100:00100000:3.0:1789760727.952734:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5083b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508915200:0@lo:13:osp-pre-0-1.0 00000001:00080000:1.0:1789760727.964625:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972247 is committed 00002000:00010000:3.0:1789760727.964778:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20189184 -> 20193280 00010000:00010000:3.0:1789760727.964792:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20189184->20193279] (req 20189184->20193279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.964805:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20189184->20193279] (req 20189184->20193279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.964814:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20189184->20193279] (req 20189184->20193279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.964821:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760727.964826:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20189184->20193279] (req 20189184->20193279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.964835:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20189184->20193279] (req 20189184->20193279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.964854:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20189184->20193279] (req 20189184->20193279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760727.964863:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208084 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20189184->20193279] (req 20189184->20193279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760727.964962:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484194432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40556us (40914us total) trans 4294972247 rc 0/0 00000100:00100000:3.0:1789760727.964972:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4958 00000100:00100000:3.0:1789760727.965529:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717203829435 time 717255321958 state 0x2 ref 0 00000100:00100000:1.0:1789760727.974851:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760727.974995:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484194560 00000100:00100000:0.0:1789760727.975004:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484194560 00000100:00100000:0.0:1789760727.975094:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717203829435 time 717255321958 state 0x0 ref 1 00000100:00100000:0.0:1789760727.975111:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717255582374 time 717255321958 state 0x1 ref 1 00000100:00100000:0.0:1789760727.975145:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717255582374 time 717306075466 state 0x0 ref 1 00000100:00100000:0.0:1789760727.975162:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484194560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760727.975212:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20193280, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:0.0:1789760727.975221:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.975237:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:0.0:1789760727.975241:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760727.975247:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20193280->20197375] (req 20193280->20197375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760727.975261:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20193280->20197375] (req 20193280->20197375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760728.001759:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20193280 -> 20197376 00010000:00010000:0.0:1789760728.001778:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20193280->20197375] (req 20193280->20197375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.001792:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20193280->20197375] (req 20193280->20197375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.001803:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20193280->20197375] (req 20193280->20197375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.001811:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760728.001816:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20193280->20197375] (req 20193280->20197375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.001826:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20193280->20197375] (req 20193280->20197375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.001845:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20193280->20197375] (req 20193280->20197375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.001853:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620808b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20193280->20197375] (req 20193280->20197375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760728.001952:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484194560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26784us (27101us total) trans 4294972248 rc 0/0 00000100:00100000:0.0:1789760728.001962:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4959 00000100:00100000:0.0:1789760728.001972:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717255582374 time 717306075466 state 0x2 ref 0 00000001:00080000:3.0:1789760728.002046:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972248 is committed 00000100:00100000:1.0:1789760728.005414:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760728.005597:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484194688 00000100:00100000:2.0:1789760728.005607:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484194688 00000100:00080000:0.0:1789760728.006436:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760728.006488:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717255582374 time 717306075466 state 0x0 ref 1 00000100:00100000:2.0:1789760728.006503:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717306335882 time 717306075466 state 0x1 ref 1 00000100:00100000:2.0:1789760728.006535:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717306335882 time 717337466265 state 0x0 ref 1 00000100:00100000:2.0:1789760728.006549:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484194688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760728.006605:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20197376, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:2.0:1789760728.006615:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.006632:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:2.0:1789760728.006636:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760728.006642:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20197376->20201471] (req 20197376->20201471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.006655:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20197376->20201471] (req 20197376->20201471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760728.034281:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972249 is committed 00002000:00010000:3.0:1789760728.034509:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20197376 -> 20201472 00010000:00010000:3.0:1789760728.034524:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20197376->20201471] (req 20197376->20201471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.034537:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20197376->20201471] (req 20197376->20201471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.034546:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20197376->20201471] (req 20197376->20201471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.034552:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.034556:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20197376->20201471] (req 20197376->20201471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.034565:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20197376->20201471] (req 20197376->20201471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.034589:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20197376->20201471] (req 20197376->20201471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.034598:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208092 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20197376->20201471] (req 20197376->20201471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.034786:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484194688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28237us (29376us total) trans 4294972249 rc 0/0 00000100:00100000:3.0:1789760728.034796:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4960 00000100:00100000:3.0:1789760728.034805:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717306335882 time 717337466265 state 0x2 ref 0 00000100:00100000:1.0:1789760728.043920:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.044221:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484194816 00000100:00100000:3.0:1789760728.044230:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484194816 00000100:00100000:2.0:1789760728.046156:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717306335882 time 717337466265 state 0x0 ref 1 00000100:00100000:2.0:1789760728.046175:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717337726681 time 717337466265 state 0x1 ref 1 00000100:00100000:2.0:1789760728.046255:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717337726681 time 717377185072 state 0x0 ref 1 00000100:00100000:2.0:1789760728.046272:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484194816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760728.046332:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20201472, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:2.0:1789760728.046342:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.046358:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:2.0:1789760728.046362:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760728.046368:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20201472->20205567] (req 20201472->20205567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.046381:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20201472->20205567] (req 20201472->20205567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760728.083692:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972250 is committed 00002000:00010000:3.0:1789760728.083951:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20201472 -> 20205568 00010000:00010000:3.0:1789760728.083973:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20201472->20205567] (req 20201472->20205567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.083987:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20201472->20205567] (req 20201472->20205567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.083997:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20201472->20205567] (req 20201472->20205567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.084005:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.084010:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20201472->20205567] (req 20201472->20205567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.084026:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20201472->20205567] (req 20201472->20205567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.084052:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20201472->20205567] (req 20201472->20205567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.084060:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208099 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20201472->20205567] (req 20201472->20205567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.084295:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484194816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38022us (40376us total) trans 4294972250 rc 0/0 00000100:00100000:3.0:1789760728.084309:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4961 00000100:00100000:3.0:1789760728.084331:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717337726681 time 717377185072 state 0x2 ref 0 00000100:00100000:1.0:1789760728.090690:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.091838:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484194944 00000100:00100000:3.0:1789760728.091854:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484194944 00000100:00100000:3.0:1789760728.091939:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717337726681 time 717377185072 state 0x0 ref 1 00000100:00100000:3.0:1789760728.091959:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717377445488 time 717377185072 state 0x1 ref 1 00000100:00100000:3.0:1789760728.092009:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717377445488 time 717422939776 state 0x0 ref 1 00000100:00100000:3.0:1789760728.092025:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484194944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760728.092120:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20205568, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:3.0:1789760728.092133:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.092156:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760728.092161:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760728.092169:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20205568->20209663] (req 20205568->20209663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.092187:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20205568->20209663] (req 20205568->20209663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760728.132856:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972251 is committed 00002000:00010000:3.0:1789760728.133266:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20205568 -> 20209664 00010000:00010000:3.0:1789760728.133287:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20205568->20209663] (req 20205568->20209663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.133303:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20205568->20209663] (req 20205568->20209663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.133313:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20205568->20209663] (req 20205568->20209663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.133321:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.133327:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20205568->20209663] (req 20205568->20209663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.133336:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20205568->20209663] (req 20205568->20209663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.133361:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20205568->20209663] (req 20205568->20209663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.133370:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080a0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20205568->20209663] (req 20205568->20209663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.133550:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484194944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41521us (42867us total) trans 4294972251 rc 0/0 00000100:00100000:3.0:1789760728.133563:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4962 00000100:00100000:3.0:1789760728.133575:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717377445488 time 717422939776 state 0x2 ref 0 00000100:00100000:1.0:1789760728.137764:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760728.138664:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484195072 00000100:00100000:0.0:1789760728.138681:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484195072 00000100:00100000:2.0:1789760728.138859:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717377445488 time 717422939776 state 0x0 ref 1 00000100:00100000:2.0:1789760728.138879:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717423200192 time 717422939776 state 0x1 ref 1 00000100:00100000:2.0:1789760728.138949:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717423200192 time 717469877778 state 0x0 ref 1 00000100:00100000:2.0:1789760728.138964:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484195072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760728.139040:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20209664, mtime: 0x6aad94d7, atime: 0x0, ctime: 0x6aad94d7, blocks: 0xa007 00010000:00010000:2.0:1789760728.139053:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.139100:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:2.0:1789760728.139106:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760728.139113:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20209664->20213759] (req 20209664->20213759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.139131:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20209664->20213759] (req 20209664->20213759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760728.187550:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972252 is committed 00002000:00010000:2.0:1789760728.187667:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20209664 -> 20213760 00002000:00010000:2.0:1789760728.187672:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760727 -> 1789760728 00002000:00010000:2.0:1789760728.187675:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760727 -> 1789760728 00010000:00010000:2.0:1789760728.187696:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20209664->20213759] (req 20209664->20213759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.187716:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20209664->20213759] (req 20209664->20213759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.187728:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20209664->20213759] (req 20209664->20213759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.187736:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760728.187742:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20209664->20213759] (req 20209664->20213759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.187759:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20209664->20213759] (req 20209664->20213759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.187785:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20209664->20213759] (req 20209664->20213759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.187794:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080a7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20209664->20213759] (req 20209664->20213759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760728.187959:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484195072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48991us (50204us total) trans 4294972252 rc 0/0 00000100:00100000:2.0:1789760728.187972:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4963 00000100:00100000:2.0:1789760728.187994:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717423200192 time 717469877778 state 0x2 ref 0 00000100:00100000:1.0:1789760728.192917:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760728.193192:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484195200 00000100:00100000:2.0:1789760728.193207:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484195200 00000100:00100000:2.0:1789760728.193281:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717423200192 time 717469877778 state 0x0 ref 1 00000100:00100000:2.0:1789760728.193302:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717470138194 time 717469877778 state 0x1 ref 1 00000100:00100000:2.0:1789760728.193656:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717470138194 time 717524583868 state 0x0 ref 1 00000100:00100000:2.0:1789760728.193674:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484195200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760728.193747:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20213760, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:2.0:1789760728.193758:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.193780:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:2.0:1789760728.193786:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760728.193794:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20213760->20217855] (req 20213760->20217855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.193810:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20213760->20217855] (req 20213760->20217855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760728.258304:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972253 is committed 00002000:00010000:3.0:1789760728.258460:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20213760 -> 20217856 00010000:00010000:3.0:1789760728.258481:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20213760->20217855] (req 20213760->20217855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.258496:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20213760->20217855] (req 20213760->20217855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.258506:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20213760->20217855] (req 20213760->20217855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.258513:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.258519:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20213760->20217855] (req 20213760->20217855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.258530:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20213760->20217855] (req 20213760->20217855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.258562:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20213760->20217855] (req 20213760->20217855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.258572:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962080ae lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20213760->20217855] (req 20213760->20217855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.258703:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484195200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65025us (65792us total) trans 4294972253 rc 0/0 00000100:00100000:3.0:1789760728.258716:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4964 00000100:00100000:3.0:1789760728.258728:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717470138194 time 717524583868 state 0x2 ref 0 00000100:00100000:1.0:1789760728.263448:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.263543:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484195328 00000100:00100000:3.0:1789760728.263557:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484195328 00000100:00100000:3.0:1789760728.263635:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717470138194 time 717524583868 state 0x0 ref 1 00000100:00100000:3.0:1789760728.263653:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717524844284 time 717524583868 state 0x1 ref 1 00000100:00100000:3.0:1789760728.263759:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717524844284 time 717594688664 state 0x0 ref 1 00000100:00100000:3.0:1789760728.263775:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484195328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760728.263844:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20217856, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:3.0:1789760728.263855:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.263875:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760728.263882:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760728.263889:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20217856->20221951] (req 20217856->20221951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.263906:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20217856->20221951] (req 20217856->20221951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760728.303958:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20217856 -> 20221952 00010000:00010000:3.0:1789760728.303995:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20217856->20221951] (req 20217856->20221951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.304011:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20217856->20221951] (req 20217856->20221951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.304021:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20217856->20221951] (req 20217856->20221951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.304028:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.304034:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20217856->20221951] (req 20217856->20221951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.304045:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20217856->20221951] (req 20217856->20221951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.304090:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20217856->20221951] (req 20217856->20221951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.304104:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080b5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20217856->20221951] (req 20217856->20221951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.304773:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484195328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40992us (41330us total) trans 4294972254 rc 0/0 00000100:00100000:3.0:1789760728.304787:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4965 00000100:00100000:3.0:1789760728.304800:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717524844284 time 717594688664 state 0x2 ref 0 00000001:00080000:0.0:1789760728.304911:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972254 is committed 00000100:00100000:1.0:1789760728.309726:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.309842:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484195456 00000100:00100000:3.0:1789760728.309855:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484195456 00000100:00100000:3.0:1789760728.309933:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717524844284 time 717594688664 state 0x0 ref 1 00000100:00100000:3.0:1789760728.309953:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717594949080 time 717594688664 state 0x1 ref 1 00000100:00100000:3.0:1789760728.310055:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717594949080 time 717640984811 state 0x0 ref 1 00000100:00100000:3.0:1789760728.310127:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484195456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760728.310410:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20221952, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:3.0:1789760728.310431:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.310450:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760728.310456:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760728.310464:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20221952->20226047] (req 20221952->20226047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.310488:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20221952->20226047] (req 20221952->20226047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760728.345225:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972255 is committed 00002000:00010000:0.0:1789760728.345641:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20221952 -> 20226048 00010000:00010000:0.0:1789760728.345663:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20221952->20226047] (req 20221952->20226047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.345678:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20221952->20226047] (req 20221952->20226047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.345695:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20221952->20226047] (req 20221952->20226047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.345704:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760728.345710:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20221952->20226047] (req 20221952->20226047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.345719:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20221952->20226047] (req 20221952->20226047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.345745:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20221952->20226047] (req 20221952->20226047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.345753:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962080bc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20221952->20226047] (req 20221952->20226047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760728.345940:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484195456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35811us (36224us total) trans 4294972255 rc 0/0 00000100:00100000:0.0:1789760728.345952:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4966 00000100:00100000:0.0:1789760728.345965:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717594949080 time 717640984811 state 0x2 ref 0 00000100:00100000:2.0:1789760728.354445:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.355534:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484195584 00000100:00100000:3.0:1789760728.355550:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484195584 00000100:00100000:3.0:1789760728.355614:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717594949080 time 717640984811 state 0x0 ref 1 00000100:00100000:3.0:1789760728.355629:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717641245227 time 717640984811 state 0x1 ref 1 00000100:00100000:3.0:1789760728.355670:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717641245227 time 717686600571 state 0x0 ref 1 00000100:00100000:3.0:1789760728.355684:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484195584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760728.355743:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20226048, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:3.0:1789760728.355752:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.355772:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760728.355776:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760728.355782:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20226048->20230143] (req 20226048->20230143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.355797:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20226048->20230143] (req 20226048->20230143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760728.389515:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20226048 -> 20230144 00010000:00010000:2.0:1789760728.389535:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20226048->20230143] (req 20226048->20230143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.389549:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20226048->20230143] (req 20226048->20230143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.389558:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20226048->20230143] (req 20226048->20230143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.389567:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760728.389572:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20226048->20230143] (req 20226048->20230143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.389581:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20226048->20230143] (req 20226048->20230143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.389605:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20226048->20230143] (req 20226048->20230143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.389614:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962080c3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20226048->20230143] (req 20226048->20230143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760728.389699:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484195584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34013us (35264us total) trans 4294972256 rc 0/0 00000100:00100000:2.0:1789760728.389716:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4967 00000100:00100000:2.0:1789760728.389726:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717641245227 time 717686600571 state 0x2 ref 0 00000001:00080000:0.0:1789760728.391797:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972256 is committed 00000100:00100000:2.0:1789760728.395659:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.395833:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484195712 00000100:00100000:3.0:1789760728.395843:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484195712 00000100:00100000:3.0:1789760728.395895:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717641245227 time 717686600571 state 0x0 ref 1 00000100:00100000:3.0:1789760728.395910:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717686860987 time 717686600571 state 0x1 ref 1 00000100:00100000:3.0:1789760728.395984:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717686860987 time 717726914442 state 0x0 ref 1 00000100:00100000:3.0:1789760728.395998:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484195712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760728.396046:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20230144, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:3.0:1789760728.396054:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.396113:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760728.396118:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760728.396125:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20230144->20234239] (req 20230144->20234239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.396139:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20230144->20234239] (req 20230144->20234239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760728.425239:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972257 is committed 00002000:00010000:3.0:1789760728.425443:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20230144 -> 20234240 00010000:00010000:3.0:1789760728.425459:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20230144->20234239] (req 20230144->20234239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.425471:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20230144->20234239] (req 20230144->20234239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.425486:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20230144->20234239] (req 20230144->20234239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.425493:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.425498:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20230144->20234239] (req 20230144->20234239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.425578:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20230144->20234239] (req 20230144->20234239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.425598:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20230144->20234239] (req 20230144->20234239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.425607:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962080ca lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20230144->20234239] (req 20230144->20234239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.426381:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484195712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30378us (30725us total) trans 4294972257 rc 0/0 00000100:00100000:3.0:1789760728.426392:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4968 00000100:00100000:3.0:1789760728.426402:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717686860987 time 717726914442 state 0x2 ref 0 00000100:00100000:1.0:1789760728.434988:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760728.435143:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484195840 00000100:00100000:2.0:1789760728.435153:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484195840 00000100:00100000:2.0:1789760728.435201:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717686860987 time 717726914442 state 0x0 ref 1 00000100:00100000:2.0:1789760728.435216:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717727174858 time 717726914442 state 0x1 ref 1 00000100:00100000:2.0:1789760728.435245:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717727174858 time 717766175961 state 0x0 ref 1 00000100:00100000:2.0:1789760728.435256:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484195840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760728.435300:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20234240, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:2.0:1789760728.435308:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.435329:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:2.0:1789760728.435333:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760728.435339:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20234240->20238335] (req 20234240->20238335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.435352:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20234240->20238335] (req 20234240->20238335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760728.461530:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508915456:0@lo:41:osp-pre-1-0.0 00000100:00100000:0.0:1789760728.461596:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760728.461667:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508915456 00000100:00100000:0.0:1789760728.461674:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508915456 00000100:00100000:0.0:1789760728.461719:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 430 00000100:00100000:0.0:1789760728.461729:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46300 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508915456:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:0.0:1789760728.461766:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:0.0:1789760728.461808:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46300 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508915456:12345-0@lo:41:osp-pre-1-0.0 Request processed in 81us (211us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760728.461816:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 430 00000100:00100000:0.0:1789760728.461858:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508915456:0@lo:41:osp-pre-1-0.0 00000001:00080000:0.0:1789760728.481620:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972258 is committed 00002000:00010000:1.0:1789760728.481740:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20234240 -> 20238336 00010000:00010000:1.0:1789760728.481757:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20234240->20238335] (req 20234240->20238335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.481769:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20234240->20238335] (req 20234240->20238335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.481779:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20234240->20238335] (req 20234240->20238335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.481787:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760728.481792:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20234240->20238335] (req 20234240->20238335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.481802:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20234240->20238335] (req 20234240->20238335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.481823:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20234240->20238335] (req 20234240->20238335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.481831:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962080d1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20234240->20238335] (req 20234240->20238335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760728.481928:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484195840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46668us (46943us total) trans 4294972258 rc 0/0 00000100:00100000:1.0:1789760728.481938:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4969 00000100:00100000:1.0:1789760728.481949:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717727174858 time 717766175961 state 0x2 ref 0 00000100:00100000:0.0:1789760728.491881:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.491976:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484195968 00000100:00100000:3.0:1789760728.491986:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484195968 00000100:00100000:3.0:1789760728.492040:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717727174858 time 717766175961 state 0x0 ref 1 00000100:00100000:3.0:1789760728.492056:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717766436377 time 717766175961 state 0x1 ref 1 00000100:00100000:3.0:1789760728.492124:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717766436377 time 717823054892 state 0x0 ref 1 00000100:00100000:3.0:1789760728.492137:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484195968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760728.492184:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20238336, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:3.0:1789760728.492192:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.492208:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760728.492212:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760728.492217:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20238336->20242431] (req 20238336->20242431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.492230:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20238336->20242431] (req 20238336->20242431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760728.552447:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972259 is committed 00002000:00010000:3.0:1789760728.552599:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20238336 -> 20242432 00010000:00010000:3.0:1789760728.552614:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20238336->20242431] (req 20238336->20242431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.552627:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20238336->20242431] (req 20238336->20242431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.552636:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20238336->20242431] (req 20238336->20242431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.552644:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.552649:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20238336->20242431] (req 20238336->20242431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.552659:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20238336->20242431] (req 20238336->20242431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.552723:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20238336->20242431] (req 20238336->20242431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.552731:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080d8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20238336->20242431] (req 20238336->20242431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.552859:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484195968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60718us (60981us total) trans 4294972259 rc 0/0 00000100:00100000:3.0:1789760728.552869:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4970 00000100:00100000:3.0:1789760728.552878:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717766436377 time 717823054892 state 0x2 ref 0 00000100:00100000:3.0:1789760728.558361:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760728.558792:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484196096 00000100:00100000:0.0:1789760728.558803:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484196096 00000100:00100000:0.0:1789760728.558863:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717766436377 time 717823054892 state 0x0 ref 1 00000100:00100000:0.0:1789760728.558877:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717823315308 time 717823054892 state 0x1 ref 1 00000100:00100000:0.0:1789760728.559004:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717823315308 time 717889933810 state 0x0 ref 1 00000100:00100000:0.0:1789760728.559020:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484196096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760728.559092:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20242432, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:0.0:1789760728.559101:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.559116:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:0.0:1789760728.559121:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760728.559128:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20242432->20246527] (req 20242432->20246527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.559141:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20242432->20246527] (req 20242432->20246527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760728.590027:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972260 is committed 00002000:00010000:2.0:1789760728.590303:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20242432 -> 20246528 00010000:00010000:2.0:1789760728.590320:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20242432->20246527] (req 20242432->20246527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.590333:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20242432->20246527] (req 20242432->20246527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.590342:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20242432->20246527] (req 20242432->20246527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.590350:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760728.590354:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20242432->20246527] (req 20242432->20246527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.590364:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20242432->20246527] (req 20242432->20246527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.590385:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20242432->20246527] (req 20242432->20246527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760728.590394:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962080df lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20242432->20246527] (req 20242432->20246527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760728.590503:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484196096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31482us (32145us total) trans 4294972260 rc 0/0 00000100:00100000:2.0:1789760728.590513:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4971 00000100:00100000:2.0:1789760728.590523:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717823315308 time 717889933810 state 0x2 ref 0 00000100:00100000:1.0:1789760728.598811:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.598899:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484196224 00000100:00100000:3.0:1789760728.598908:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484196224 00000100:00100000:3.0:1789760728.598960:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717823315308 time 717889933810 state 0x0 ref 1 00000100:00100000:3.0:1789760728.598975:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717890194226 time 717889933810 state 0x1 ref 1 00000100:00100000:3.0:1789760728.599009:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717890194226 time 717929939189 state 0x0 ref 1 00000100:00100000:3.0:1789760728.599021:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484196224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760728.599090:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20246528, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:3.0:1789760728.599099:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.599115:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:3.0:1789760728.599120:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760728.599126:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20246528->20250623] (req 20246528->20250623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.599141:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20246528->20250623] (req 20246528->20250623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760728.634464:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972261 is committed 00002000:00010000:3.0:1789760728.634816:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20246528 -> 20250624 00010000:00010000:3.0:1789760728.634831:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20246528->20250623] (req 20246528->20250623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.634843:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20246528->20250623] (req 20246528->20250623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.634853:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20246528->20250623] (req 20246528->20250623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.634861:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.634865:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20246528->20250623] (req 20246528->20250623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.634874:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20246528->20250623] (req 20246528->20250623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.634894:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20246528->20250623] (req 20246528->20250623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.634902:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962080e6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20246528->20250623] (req 20246528->20250623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.635031:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484196224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36007us (36224us total) trans 4294972261 rc 0/0 00000100:00100000:3.0:1789760728.635042:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4972 00000100:00100000:3.0:1789760728.635151:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717890194226 time 717929939189 state 0x2 ref 0 00000100:00100000:0.0:1789760728.638716:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760728.638817:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484196352 00000100:00100000:1.0:1789760728.638827:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484196352 00000100:00100000:1.0:1789760728.638880:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717890194226 time 717929939189 state 0x0 ref 1 00000100:00100000:1.0:1789760728.638895:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717930199605 time 717929939189 state 0x1 ref 1 00000100:00100000:1.0:1789760728.638964:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717930199605 time 717969894920 state 0x0 ref 1 00000100:00100000:1.0:1789760728.638977:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484196352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760728.639023:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20250624, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:1.0:1789760728.639031:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.639056:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:1.0:1789760728.639062:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760728.639102:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20250624->20254719] (req 20250624->20254719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.639117:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20250624->20254719] (req 20250624->20254719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760728.663512:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20250624 -> 20254720 00010000:00010000:1.0:1789760728.663532:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20250624->20254719] (req 20250624->20254719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.663545:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20250624->20254719] (req 20250624->20254719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.663553:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20250624->20254719] (req 20250624->20254719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.663560:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760728.663565:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20250624->20254719] (req 20250624->20254719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.663574:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20250624->20254719] (req 20250624->20254719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.663591:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20250624->20254719] (req 20250624->20254719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.663598:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962080ed lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20250624->20254719] (req 20250624->20254719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760728.663684:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484196352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24706us (24973us total) trans 4294972262 rc 0/0 00000100:00100000:1.0:1789760728.663697:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4973 00000100:00100000:1.0:1789760728.663705:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717930199605 time 717969894920 state 0x2 ref 0 00000001:00080000:3.0:1789760728.663773:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972262 is committed 00000100:00100000:2.0:1789760728.667744:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760728.667814:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484196480 00000100:00100000:1.0:1789760728.667824:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484196480 00000100:00100000:1.0:1789760728.667875:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717930199605 time 717969894920 state 0x0 ref 1 00000100:00100000:1.0:1789760728.667901:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717970155336 time 717969894920 state 0x1 ref 1 00000100:00100000:1.0:1789760728.667926:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717970155336 time 717998857102 state 0x0 ref 1 00000100:00100000:1.0:1789760728.667939:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484196480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760728.667987:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20254720, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:1.0:1789760728.667997:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.668013:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=717s v=(0 0) 00010000:00010000:1.0:1789760728.668017:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760728.668023:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20254720->20258815] (req 20254720->20258815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.668036:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20254720->20258815] (req 20254720->20258815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760728.695039:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972263 is committed 00002000:00010000:1.0:1789760728.695177:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20254720 -> 20258816 00010000:00010000:1.0:1789760728.695303:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20254720->20258815] (req 20254720->20258815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.695319:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20254720->20258815] (req 20254720->20258815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.695328:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20254720->20258815] (req 20254720->20258815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.695336:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760728.695341:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20254720->20258815] (req 20254720->20258815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.695352:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20254720->20258815] (req 20254720->20258815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.695376:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20254720->20258815] (req 20254720->20258815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.695384:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962080f4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20254720->20258815] (req 20254720->20258815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760728.695664:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484196480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27722us (27923us total) trans 4294972263 rc 0/0 00000100:00100000:1.0:1789760728.695675:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4974 00000100:00100000:1.0:1789760728.695685:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717970155336 time 717998857102 state 0x2 ref 0 00000100:00100000:2.0:1789760728.701036:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760728.701170:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484196608 00000100:00100000:0.0:1789760728.701180:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484196608 00000100:00100000:0.0:1789760728.701350:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717970155336 time 717998857102 state 0x0 ref 1 00000100:00100000:0.0:1789760728.701366:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717999117518 time 717998857102 state 0x1 ref 1 00000100:00100000:0.0:1789760728.701387:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717999117518 time 718032318679 state 0x0 ref 1 00000100:00100000:0.0:1789760728.701401:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484196608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760728.701458:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20258816, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:0.0:1789760728.701466:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.701482:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:0.0:1789760728.701486:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760728.701493:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20258816->20262911] (req 20258816->20262911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.701577:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20258816->20262911] (req 20258816->20262911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760728.740347:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972264 is committed 00002000:00010000:0.0:1789760728.740655:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20258816 -> 20262912 00010000:00010000:0.0:1789760728.740670:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20258816->20262911] (req 20258816->20262911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.740682:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20258816->20262911] (req 20258816->20262911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.740691:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20258816->20262911] (req 20258816->20262911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.740699:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760728.740703:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20258816->20262911] (req 20258816->20262911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.740711:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20258816->20262911] (req 20258816->20262911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.740728:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20258816->20262911] (req 20258816->20262911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.740736:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962080fb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20258816->20262911] (req 20258816->20262911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760728.740824:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484196608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39423us (39794us total) trans 4294972264 rc 0/0 00000100:00100000:0.0:1789760728.740836:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4975 00000100:00100000:0.0:1789760728.740846:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717999117518 time 718032318679 state 0x2 ref 0 00000100:00100000:2.0:1789760728.745249:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.745328:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484196736 00000100:00100000:3.0:1789760728.745338:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484196736 00000100:00100000:3.0:1789760728.745390:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 717999117518 time 718032318679 state 0x0 ref 1 00000100:00100000:3.0:1789760728.745404:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718032579095 time 718032318679 state 0x1 ref 1 00000100:00100000:3.0:1789760728.745439:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718032579095 time 718076370132 state 0x0 ref 1 00000100:00100000:3.0:1789760728.745451:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484196736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760728.745495:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20262912, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:3.0:1789760728.745511:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.745526:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:3.0:1789760728.745530:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760728.745536:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20262912->20267007] (req 20262912->20267007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.745549:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20262912->20267007] (req 20262912->20267007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760728.810709:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972265 is committed 00002000:00010000:3.0:1789760728.811654:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20262912 -> 20267008 00010000:00010000:3.0:1789760728.811670:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20262912->20267007] (req 20262912->20267007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.811685:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20262912->20267007] (req 20262912->20267007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.811697:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20262912->20267007] (req 20262912->20267007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.811703:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.811708:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20262912->20267007] (req 20262912->20267007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.811717:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20262912->20267007] (req 20262912->20267007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.811739:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20262912->20267007] (req 20262912->20267007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.811746:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208102 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20262912->20267007] (req 20262912->20267007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.811856:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484196736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66402us (66611us total) trans 4294972265 rc 0/0 00000100:00100000:3.0:1789760728.811866:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4976 00000100:00100000:3.0:1789760728.811876:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718032579095 time 718076370132 state 0x2 ref 0 00000100:00100000:3.0:1789760728.815024:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760728.815141:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484196864 00000100:00100000:1.0:1789760728.815151:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484196864 00000100:00100000:1.0:1789760728.815204:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718032579095 time 718076370132 state 0x0 ref 1 00000100:00100000:1.0:1789760728.815219:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718076630548 time 718076370132 state 0x1 ref 1 00000100:00100000:1.0:1789760728.815258:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718076630548 time 718146189191 state 0x0 ref 1 00000100:00100000:1.0:1789760728.815269:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484196864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760728.815314:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20267008, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:1.0:1789760728.815324:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.815339:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:1.0:1789760728.815343:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760728.815348:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20267008->20271103] (req 20267008->20271103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.815362:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20267008->20271103] (req 20267008->20271103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760728.861551:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972266 is committed 00002000:00010000:1.0:1789760728.861732:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20267008 -> 20271104 00010000:00010000:1.0:1789760728.861748:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20267008->20271103] (req 20267008->20271103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.862324:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20267008->20271103] (req 20267008->20271103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.862340:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20267008->20271103] (req 20267008->20271103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.862347:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760728.862352:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20267008->20271103] (req 20267008->20271103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.862361:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20267008->20271103] (req 20267008->20271103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.862381:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20267008->20271103] (req 20267008->20271103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.862388:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208109 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20267008->20271103] (req 20267008->20271103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760728.862519:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484196864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47247us (47498us total) trans 4294972266 rc 0/0 00000100:00100000:1.0:1789760728.862529:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4977 00000100:00100000:1.0:1789760728.862539:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718076630548 time 718146189191 state 0x2 ref 0 00000100:00100000:2.0:1789760728.874390:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760728.874867:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484196992 00000100:00100000:1.0:1789760728.874879:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484196992 00000100:00100000:1.0:1789760728.874937:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718076630548 time 718146189191 state 0x0 ref 1 00000100:00100000:1.0:1789760728.874950:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718146449607 time 718146189191 state 0x1 ref 1 00000100:00100000:1.0:1789760728.875032:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718146449607 time 718205961614 state 0x0 ref 1 00000100:00100000:1.0:1789760728.875046:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484196992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760728.875134:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20271104, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:1.0:1789760728.875143:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.875158:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:1.0:1789760728.875163:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760728.875169:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20271104->20275199] (req 20271104->20275199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760728.875184:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20271104->20275199] (req 20271104->20275199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760728.922245:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972267 is committed 00002000:00010000:3.0:1789760728.922433:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20271104 -> 20275200 00010000:00010000:3.0:1789760728.922450:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20271104->20275199] (req 20271104->20275199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.922463:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20271104->20275199] (req 20271104->20275199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.922473:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20271104->20275199] (req 20271104->20275199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.922481:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760728.922486:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20271104->20275199] (req 20271104->20275199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.922494:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20271104->20275199] (req 20271104->20275199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.922529:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20271104->20275199] (req 20271104->20275199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.922537:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208110 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20271104->20275199] (req 20271104->20275199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760728.922629:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484196992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47580us (48245us total) trans 4294972267 rc 0/0 00000100:00100000:3.0:1789760728.922639:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4978 00000100:00100000:3.0:1789760728.922649:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718146449607 time 718205961614 state 0x2 ref 0 00000100:00100000:2.0:1789760728.926478:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760728.927870:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484197120 00000100:00100000:0.0:1789760728.927882:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484197120 00000100:00100000:0.0:1789760728.927949:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718146449607 time 718205961614 state 0x0 ref 1 00000100:00100000:0.0:1789760728.927963:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718206222030 time 718205961614 state 0x1 ref 1 00000100:00100000:0.0:1789760728.927993:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718206222030 time 718258924496 state 0x0 ref 1 00000100:00100000:0.0:1789760728.928006:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484197120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760728.928050:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20275200, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:0.0:1789760728.928058:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.928114:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:0.0:1789760728.928118:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760728.928125:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20275200->20279295] (req 20275200->20279295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.928138:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20275200->20279295] (req 20275200->20279295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.972257:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:0.0:1789760728.981870:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20275200 -> 20279296 00010000:00010000:0.0:1789760728.981891:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20275200->20279295] (req 20275200->20279295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.981905:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20275200->20279295] (req 20275200->20279295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.981926:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20275200->20279295] (req 20275200->20279295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.981935:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760728.981940:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20275200->20279295] (req 20275200->20279295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.981949:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20275200->20279295] (req 20275200->20279295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.981970:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20275200->20279295] (req 20275200->20279295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760728.981978:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208117 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20275200->20279295] (req 20275200->20279295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760728.982118:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484197120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54110us (55643us total) trans 4294972268 rc 0/0 00000100:00100000:0.0:1789760728.982133:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4979 00000100:00100000:0.0:1789760728.982143:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718206222030 time 718258924496 state 0x2 ref 0 00000001:00080000:2.0:1789760728.983025:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972268 is committed 00000100:00100000:2.0:1789760728.986935:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760728.987023:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484197248 00000100:00100000:3.0:1789760728.987032:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484197248 00000100:00100000:3.0:1789760728.987117:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718206222030 time 718258924496 state 0x0 ref 1 00000100:00100000:3.0:1789760728.987133:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718259184912 time 718258924496 state 0x1 ref 1 00000100:00100000:3.0:1789760728.987155:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718259184912 time 718318086780 state 0x0 ref 1 00000100:00100000:3.0:1789760728.987168:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484197248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760728.987224:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20279296, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:3.0:1789760728.987233:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.987253:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:3.0:1789760728.987258:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760728.987264:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20279296->20283391] (req 20279296->20283391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760728.987278:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20279296->20283391] (req 20279296->20283391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760729.039785:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972269 is committed 00002000:00010000:0.0:1789760729.040202:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20279296 -> 20283392 00010000:00010000:0.0:1789760729.040218:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20279296->20283391] (req 20279296->20283391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.040231:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20279296->20283391] (req 20279296->20283391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.040240:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20279296->20283391] (req 20279296->20283391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.040246:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760729.040251:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20279296->20283391] (req 20279296->20283391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.040259:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20279296->20283391] (req 20279296->20283391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.040279:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20279296->20283391] (req 20279296->20283391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.040286:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620811e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20279296->20283391] (req 20279296->20283391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760729.040374:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484197248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53204us (53443us total) trans 4294972269 rc 0/0 00000100:00100000:0.0:1789760729.040384:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4980 00000100:00100000:0.0:1789760729.040394:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718259184912 time 718318086780 state 0x2 ref 0 00000100:00100000:3.0:1789760729.043916:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760729.044152:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484197376 00000100:00100000:1.0:1789760729.044161:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484197376 00000100:00100000:1.0:1789760729.044221:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718259184912 time 718318086780 state 0x0 ref 1 00000100:00100000:1.0:1789760729.044237:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718318347196 time 718318086780 state 0x1 ref 1 00000100:00100000:1.0:1789760729.044263:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718318347196 time 718375194589 state 0x0 ref 1 00000100:00100000:1.0:1789760729.044282:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484197376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760729.044326:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20283392, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:1.0:1789760729.044335:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.044351:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:1.0:1789760729.044355:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760729.044361:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20283392->20287487] (req 20283392->20287487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.044375:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20283392->20287487] (req 20283392->20287487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760729.094418:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972270 is committed 00002000:00010000:1.0:1789760729.094637:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20283392 -> 20287488 00010000:00010000:1.0:1789760729.094654:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20283392->20287487] (req 20283392->20287487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.094667:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20283392->20287487] (req 20283392->20287487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.094676:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20283392->20287487] (req 20283392->20287487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.094682:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.094687:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20283392->20287487] (req 20283392->20287487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.094696:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20283392->20287487] (req 20283392->20287487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.094787:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20283392->20287487] (req 20283392->20287487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.094795:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208125 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20283392->20287487] (req 20283392->20287487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.094924:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484197376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50639us (51013us total) trans 4294972270 rc 0/0 00000100:00100000:1.0:1789760729.094934:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4981 00000100:00100000:1.0:1789760729.094943:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718318347196 time 718375194589 state 0x2 ref 0 00000100:00100000:2.0:1789760729.098366:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760729.098462:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484197504 00000100:00100000:1.0:1789760729.098595:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484197504 00000100:00100000:1.0:1789760729.098645:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718318347196 time 718375194589 state 0x0 ref 1 00000100:00100000:1.0:1789760729.098660:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718375455005 time 718375194589 state 0x1 ref 1 00000100:00100000:1.0:1789760729.098787:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718375455005 time 718429717414 state 0x0 ref 1 00000100:00100000:1.0:1789760729.098800:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484197504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760729.098846:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20287488, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:1.0:1789760729.098855:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.098871:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:1.0:1789760729.098875:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760729.098882:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20287488->20291583] (req 20287488->20291583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.098895:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20287488->20291583] (req 20287488->20291583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760729.131833:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972271 is committed 00002000:00010000:1.0:1789760729.131936:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20287488 -> 20291584 00010000:00010000:1.0:1789760729.131951:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20287488->20291583] (req 20287488->20291583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.131964:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20287488->20291583] (req 20287488->20291583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.131974:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20287488->20291583] (req 20287488->20291583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.131982:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.131987:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20287488->20291583] (req 20287488->20291583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.131996:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20287488->20291583] (req 20287488->20291583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.132015:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20287488->20291583] (req 20287488->20291583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.132021:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620812c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20287488->20291583] (req 20287488->20291583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.132147:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484197504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33345us (33785us total) trans 4294972271 rc 0/0 00000100:00100000:1.0:1789760729.132159:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4982 00000100:00100000:1.0:1789760729.132168:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718375455005 time 718429717414 state 0x2 ref 0 00000100:00100000:3.0:1789760729.136009:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760729.136779:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484197632 00000100:00100000:1.0:1789760729.136790:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484197632 00000100:00100000:1.0:1789760729.136848:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718375455005 time 718429717414 state 0x0 ref 1 00000100:00100000:1.0:1789760729.136863:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718429977830 time 718429717414 state 0x1 ref 1 00000100:00100000:1.0:1789760729.136885:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718429977830 time 718467816056 state 0x0 ref 1 00000100:00100000:1.0:1789760729.136896:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484197632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760729.136946:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20291584, mtime: 0x6aad94d8, atime: 0x0, ctime: 0x6aad94d8, blocks: 0xa007 00010000:00010000:1.0:1789760729.136955:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.136971:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:1.0:1789760729.136975:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760729.136981:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20291584->20295679] (req 20291584->20295679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.136997:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20291584->20295679] (req 20291584->20295679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760729.168428:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972272 is committed 00002000:00010000:2.0:1789760729.168597:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20291584 -> 20295680 00002000:00010000:2.0:1789760729.168602:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760728 -> 1789760729 00002000:00010000:2.0:1789760729.168605:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760728 -> 1789760729 00010000:00010000:2.0:1789760729.168617:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20291584->20295679] (req 20291584->20295679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.168630:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20291584->20295679] (req 20291584->20295679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.168640:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20291584->20295679] (req 20291584->20295679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.168647:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760729.168651:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20291584->20295679] (req 20291584->20295679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.168661:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20291584->20295679] (req 20291584->20295679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.168679:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20291584->20295679] (req 20291584->20295679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.168686:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208133 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20291584->20295679] (req 20291584->20295679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760729.168879:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484197632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31980us (32875us total) trans 4294972272 rc 0/0 00000100:00100000:2.0:1789760729.168890:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4983 00000100:00100000:2.0:1789760729.168900:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718429977830 time 718467816056 state 0x2 ref 0 00000100:00100000:1.0:1789760729.175454:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760729.175528:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484197760 00000100:00100000:3.0:1789760729.175537:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484197760 00000100:00100000:3.0:1789760729.175590:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718429977830 time 718467816056 state 0x0 ref 1 00000100:00100000:3.0:1789760729.175605:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718468076472 time 718467816056 state 0x1 ref 1 00000100:00100000:3.0:1789760729.175671:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718468076472 time 718506601594 state 0x0 ref 1 00000100:00100000:3.0:1789760729.175684:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7071450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484197760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760729.175742:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20295680, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:3.0:1789760729.175751:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.175776:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:3.0:1789760729.175781:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760729.175786:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20295680->20299775] (req 20295680->20299775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.175802:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20295680->20299775] (req 20295680->20299775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760729.203495:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972273 is committed 00002000:00010000:2.0:1789760729.203642:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20295680 -> 20299776 00010000:00010000:2.0:1789760729.203658:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20295680->20299775] (req 20295680->20299775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.203676:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20295680->20299775] (req 20295680->20299775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.203685:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20295680->20299775] (req 20295680->20299775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.203692:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760729.203697:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20295680->20299775] (req 20295680->20299775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.203706:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20295680->20299775] (req 20295680->20299775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.203726:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20295680->20299775] (req 20295680->20299775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.203734:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620813a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20295680->20299775] (req 20295680->20299775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760729.203819:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7071450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484197760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28132us (28371us total) trans 4294972273 rc 0/0 00000100:00100000:2.0:1789760729.203830:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4984 00000100:00100000:2.0:1789760729.203839:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718468076472 time 718506601594 state 0x2 ref 0 00000100:00100000:1.0:1789760729.207833:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760729.207923:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484197888 00000100:00100000:0.0:1789760729.207933:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484197888 00000100:00100000:0.0:1789760729.207987:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718468076472 time 718506601594 state 0x0 ref 1 00000100:00100000:0.0:1789760729.208001:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718506862010 time 718506601594 state 0x1 ref 1 00000100:00100000:0.0:1789760729.208024:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718506862010 time 718538955442 state 0x0 ref 1 00000100:00100000:0.0:1789760729.208035:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484197888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760729.208107:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20299776, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:0.0:1789760729.208116:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.208131:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:0.0:1789760729.208135:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760729.208140:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20299776->20303871] (req 20299776->20303871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.208158:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20299776->20303871] (req 20299776->20303871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760729.233670:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20299776 -> 20303872 00010000:00010000:2.0:1789760729.233690:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20299776->20303871] (req 20299776->20303871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.233703:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20299776->20303871] (req 20299776->20303871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.233712:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20299776->20303871] (req 20299776->20303871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.233719:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760729.233724:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20299776->20303871] (req 20299776->20303871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.233733:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20299776->20303871] (req 20299776->20303871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.233751:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20299776->20303871] (req 20299776->20303871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.233759:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208141 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20299776->20303871] (req 20299776->20303871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760729.233851:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484197888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25813us (26023us total) trans 4294972274 rc 0/0 00000100:00100000:2.0:1789760729.233861:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4985 00000100:00100000:2.0:1789760729.233872:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718506862010 time 718538955442 state 0x2 ref 0 00000001:00080000:3.0:1789760729.233949:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972274 is committed 00000100:00100000:3.0:1789760729.237330:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760729.237430:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484198016 00000100:00100000:1.0:1789760729.237440:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484198016 00000100:00100000:1.0:1789760729.237503:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718506862010 time 718538955442 state 0x0 ref 1 00000100:00100000:1.0:1789760729.237519:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718539215858 time 718538955442 state 0x1 ref 1 00000100:00100000:1.0:1789760729.237546:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718539215858 time 718568477872 state 0x0 ref 1 00000100:00100000:1.0:1789760729.237559:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760729.237605:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20303872, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:1.0:1789760729.237614:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.237630:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:1.0:1789760729.237635:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760729.237641:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20303872->20307967] (req 20303872->20307967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.237656:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20303872->20307967] (req 20303872->20307967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760729.264879:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20303872 -> 20307968 00010000:00010000:2.0:1789760729.264899:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20303872->20307967] (req 20303872->20307967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.264912:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20303872->20307967] (req 20303872->20307967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.264920:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20303872->20307967] (req 20303872->20307967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.264928:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760729.264933:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20303872->20307967] (req 20303872->20307967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.264943:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20303872->20307967] (req 20303872->20307967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.264962:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20303872->20307967] (req 20303872->20307967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.264970:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208148 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20303872->20307967] (req 20303872->20307967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760729.264979:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972275 is committed 00000100:00100000:2.0:1789760729.265053:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27491us (27726us total) trans 4294972275 rc 0/0 00000100:00100000:2.0:1789760729.265136:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4986 00000100:00100000:2.0:1789760729.265146:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718539215858 time 718568477872 state 0x2 ref 0 00000100:00100000:0.0:1789760729.268372:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760729.268462:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484198144 00000100:00100000:2.0:1789760729.268472:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484198144 00000100:00100000:2.0:1789760729.268652:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718539215858 time 718568477872 state 0x0 ref 1 00000100:00100000:2.0:1789760729.268668:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718568738288 time 718568477872 state 0x1 ref 1 00000100:00100000:2.0:1789760729.268698:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718568738288 time 718599629160 state 0x0 ref 1 00000100:00100000:2.0:1789760729.268711:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760729.268759:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20307968, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:2.0:1789760729.268768:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.268788:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:2.0:1789760729.268792:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760729.268798:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20307968->20312063] (req 20307968->20312063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.268821:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20307968->20312063] (req 20307968->20312063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760729.295131:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972276 is committed 00002000:00010000:1.0:1789760729.295287:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20307968 -> 20312064 00010000:00010000:1.0:1789760729.295302:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20307968->20312063] (req 20307968->20312063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.295314:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20307968->20312063] (req 20307968->20312063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.295324:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20307968->20312063] (req 20307968->20312063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.295331:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.295336:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20307968->20312063] (req 20307968->20312063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.295345:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20307968->20312063] (req 20307968->20312063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.295366:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20307968->20312063] (req 20307968->20312063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.295374:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620814f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20307968->20312063] (req 20307968->20312063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.295473:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26760us (27105us total) trans 4294972276 rc 0/0 00000100:00100000:1.0:1789760729.295483:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4987 00000100:00100000:1.0:1789760729.295493:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718568738288 time 718599629160 state 0x2 ref 0 00000100:00100000:1.0:1789760729.299282:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760729.299501:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484198272 00000100:00100000:2.0:1789760729.299511:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484198272 00000100:00100000:2.0:1789760729.299564:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718568738288 time 718599629160 state 0x0 ref 1 00000100:00100000:2.0:1789760729.299579:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718599889576 time 718599629160 state 0x1 ref 1 00000100:00100000:2.0:1789760729.299617:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718599889576 time 718630547613 state 0x0 ref 1 00000100:00100000:2.0:1789760729.299629:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760729.299676:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20312064, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:2.0:1789760729.299684:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.299699:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:2.0:1789760729.299703:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760729.299708:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20312064->20316159] (req 20312064->20316159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.299721:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20312064->20316159] (req 20312064->20316159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760729.352899:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972277 is committed 00002000:00010000:1.0:1789760729.353015:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20312064 -> 20316160 00010000:00010000:1.0:1789760729.353029:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20312064->20316159] (req 20312064->20316159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.353043:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20312064->20316159] (req 20312064->20316159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.353052:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20312064->20316159] (req 20312064->20316159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.353059:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.353091:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20312064->20316159] (req 20312064->20316159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.353102:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20312064->20316159] (req 20312064->20316159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.353120:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20312064->20316159] (req 20312064->20316159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.353128:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208156 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20312064->20316159] (req 20312064->20316159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.353210:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53577us (53932us total) trans 4294972277 rc 0/0 00000100:00100000:1.0:1789760729.353220:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4988 00000100:00100000:1.0:1789760729.353229:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718599889576 time 718630547613 state 0x2 ref 0 00000100:00100000:2.0:1789760729.356531:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760729.356867:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484198400 00000100:00100000:0.0:1789760729.356877:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484198400 00000100:00100000:0.0:1789760729.356933:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718599889576 time 718630547613 state 0x0 ref 1 00000100:00100000:0.0:1789760729.356947:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718630808029 time 718630547613 state 0x1 ref 1 00000100:00100000:0.0:1789760729.356989:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718630808029 time 718687919855 state 0x0 ref 1 00000100:00100000:0.0:1789760729.357001:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760729.357046:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20316160, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:0.0:1789760729.357055:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.357102:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:0.0:1789760729.357107:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760729.357113:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20316160->20320255] (req 20316160->20320255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.357126:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20316160->20320255] (req 20316160->20320255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760729.383444:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20316160 -> 20320256 00010000:00010000:2.0:1789760729.383464:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20316160->20320255] (req 20316160->20320255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.383477:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20316160->20320255] (req 20316160->20320255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.383485:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20316160->20320255] (req 20316160->20320255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.383497:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760729.383502:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20316160->20320255] (req 20316160->20320255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.383511:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20316160->20320255] (req 20316160->20320255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.383535:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20316160->20320255] (req 20316160->20320255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.383544:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620815d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20316160->20320255] (req 20316160->20320255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760729.383641:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484198400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26638us (27113us total) trans 4294972278 rc 0/0 00000100:00100000:2.0:1789760729.383651:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4989 00000100:00100000:2.0:1789760729.383661:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718630808029 time 718687919855 state 0x2 ref 0 00000001:00080000:1.0:1789760729.383734:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972278 is committed 00000100:00100000:0.0:1789760729.392563:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760729.392695:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484198528 00000100:00100000:3.0:1789760729.392715:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484198528 00000100:00100000:3.0:1789760729.392766:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718630808029 time 718687919855 state 0x0 ref 1 00000100:00100000:3.0:1789760729.392781:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718688180271 time 718687919855 state 0x1 ref 1 00000100:00100000:3.0:1789760729.392815:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718688180271 time 718723746350 state 0x0 ref 1 00000100:00100000:3.0:1789760729.392829:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760729.392874:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20320256, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:3.0:1789760729.392882:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.392897:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:3.0:1789760729.392902:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760729.392907:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20320256->20324351] (req 20320256->20324351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.392926:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20320256->20324351] (req 20320256->20324351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760729.426814:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972279 is committed 00002000:00010000:3.0:1789760729.426930:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20320256 -> 20324352 00010000:00010000:3.0:1789760729.426945:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20320256->20324351] (req 20320256->20324351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.426959:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20320256->20324351] (req 20320256->20324351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.426968:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20320256->20324351] (req 20320256->20324351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.426976:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760729.426980:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20320256->20324351] (req 20320256->20324351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.426989:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20320256->20324351] (req 20320256->20324351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.427092:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20320256->20324351] (req 20320256->20324351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.427101:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208164 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20320256->20324351] (req 20320256->20324351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760729.427276:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34444us (34716us total) trans 4294972279 rc 0/0 00000100:00100000:3.0:1789760729.427286:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4990 00000100:00100000:3.0:1789760729.427295:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718688180271 time 718723746350 state 0x2 ref 0 00000100:00100000:0.0:1789760729.430784:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760729.430858:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484198656 00000100:00100000:2.0:1789760729.430867:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484198656 00000100:00100000:2.0:1789760729.430914:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718688180271 time 718723746350 state 0x0 ref 1 00000100:00100000:2.0:1789760729.430926:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718724006766 time 718723746350 state 0x1 ref 1 00000100:00100000:2.0:1789760729.430954:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718724006766 time 718761885025 state 0x0 ref 1 00000100:00100000:2.0:1789760729.430966:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760729.431156:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20324352, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:2.0:1789760729.431165:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.431181:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:2.0:1789760729.431185:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760729.431255:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20324352->20328447] (req 20324352->20328447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.431269:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20324352->20328447] (req 20324352->20328447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760729.465381:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20324352 -> 20328448 00010000:00010000:1.0:1789760729.465401:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20324352->20328447] (req 20324352->20328447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.465425:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20324352->20328447] (req 20324352->20328447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.465435:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20324352->20328447] (req 20324352->20328447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.465442:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.465446:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20324352->20328447] (req 20324352->20328447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.465454:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20324352->20328447] (req 20324352->20328447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.465473:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20324352->20328447] (req 20324352->20328447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.465479:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620816b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20324352->20328447] (req 20324352->20328447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.465568:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484198656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34600us (34787us total) trans 4294972280 rc 0/0 00000100:00100000:1.0:1789760729.465583:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4991 00000100:00100000:1.0:1789760729.465591:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718724006766 time 718761885025 state 0x2 ref 0 00000001:00080000:0.0:1789760729.465669:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972280 is committed 00000100:00100000:2.0:1789760729.469668:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760729.469771:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484198784 00000100:00100000:0.0:1789760729.469781:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484198784 00000100:00100000:0.0:1789760729.469833:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718724006766 time 718761885025 state 0x0 ref 1 00000100:00100000:0.0:1789760729.469847:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718762145441 time 718761885025 state 0x1 ref 1 00000100:00100000:0.0:1789760729.469906:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718762145441 time 718800836840 state 0x0 ref 1 00000100:00100000:0.0:1789760729.469920:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484198784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760729.469972:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20328448, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:0.0:1789760729.469981:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.469996:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:0.0:1789760729.470000:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760729.470006:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20328448->20332543] (req 20328448->20332543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.470019:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20328448->20332543] (req 20328448->20332543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760729.497824:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20328448 -> 20332544 00010000:00010000:0.0:1789760729.497844:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20328448->20332543] (req 20328448->20332543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.497866:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20328448->20332543] (req 20328448->20332543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.497876:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20328448->20332543] (req 20328448->20332543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.497884:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760729.497889:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20328448->20332543] (req 20328448->20332543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.497899:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20328448->20332543] (req 20328448->20332543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.497917:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20328448->20332543] (req 20328448->20332543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.497924:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208172 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20328448->20332543] (req 20328448->20332543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760729.498014:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484198784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28094us (28351us total) trans 4294972281 rc 0/0 00000100:00100000:0.0:1789760729.498026:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4992 00000100:00100000:0.0:1789760729.498036:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718762145441 time 718800836840 state 0x2 ref 0 00000001:00080000:2.0:1789760729.498504:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972281 is committed 00000100:00100000:2.0:1789760729.500910:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760729.500988:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484198912 00000100:00100000:0.0:1789760729.500997:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484198912 00000100:00100000:0.0:1789760729.501053:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718762145441 time 718800836840 state 0x0 ref 1 00000100:00100000:0.0:1789760729.501101:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718801097256 time 718800836840 state 0x1 ref 1 00000100:00100000:0.0:1789760729.501121:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718801097256 time 718832052413 state 0x0 ref 1 00000100:00100000:0.0:1789760729.501133:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484198912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760729.501178:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20332544, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:0.0:1789760729.501187:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.501210:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:0.0:1789760729.501215:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760729.501221:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20332544->20336639] (req 20332544->20336639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.501234:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20332544->20336639] (req 20332544->20336639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760729.530900:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20332544 -> 20336640 00010000:00010000:0.0:1789760729.530920:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20332544->20336639] (req 20332544->20336639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.530932:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20332544->20336639] (req 20332544->20336639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.530940:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20332544->20336639] (req 20332544->20336639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.530947:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760729.530952:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20332544->20336639] (req 20332544->20336639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.530961:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20332544->20336639] (req 20332544->20336639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.530980:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20332544->20336639] (req 20332544->20336639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.530987:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208179 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20332544->20336639] (req 20332544->20336639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760729.531102:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484198912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29967us (30195us total) trans 4294972282 rc 0/0 00000100:00100000:0.0:1789760729.531112:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4993 00000100:00100000:0.0:1789760729.531121:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718801097256 time 718832052413 state 0x2 ref 0 00000001:00080000:2.0:1789760729.531228:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972282 is committed 00000100:00100000:1.0:1789760729.535781:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760729.535860:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484199040 00000100:00100000:3.0:1789760729.535869:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484199040 00000100:00100000:3.0:1789760729.535921:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718801097256 time 718832052413 state 0x0 ref 1 00000100:00100000:3.0:1789760729.535935:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718832312829 time 718832052413 state 0x1 ref 1 00000100:00100000:3.0:1789760729.535964:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718832312829 time 718866895082 state 0x0 ref 1 00000100:00100000:3.0:1789760729.535976:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760729.536021:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20336640, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:3.0:1789760729.536030:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.536046:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:3.0:1789760729.536050:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760729.536057:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20336640->20340735] (req 20336640->20340735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.536103:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20336640->20340735] (req 20336640->20340735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760729.567805:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972283 is committed 00002000:00010000:1.0:1789760729.567910:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20336640 -> 20340736 00010000:00010000:1.0:1789760729.567926:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20336640->20340735] (req 20336640->20340735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.567939:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20336640->20340735] (req 20336640->20340735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.567948:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20336640->20340735] (req 20336640->20340735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.567955:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.567960:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20336640->20340735] (req 20336640->20340735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.567967:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20336640->20340735] (req 20336640->20340735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.567986:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20336640->20340735] (req 20336640->20340735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.567994:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208180 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20336640->20340735] (req 20336640->20340735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.568125:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32146us (32348us total) trans 4294972283 rc 0/0 00000100:00100000:1.0:1789760729.568137:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4994 00000100:00100000:1.0:1789760729.568146:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718832312829 time 718866895082 state 0x2 ref 0 00000100:00100000:1.0:1789760729.571783:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760729.571999:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484199168 00000100:00100000:2.0:1789760729.572010:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484199168 00000100:00100000:2.0:1789760729.572061:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718832312829 time 718866895082 state 0x0 ref 1 00000100:00100000:2.0:1789760729.572118:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718867155498 time 718866895082 state 0x1 ref 1 00000100:00100000:2.0:1789760729.572139:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718867155498 time 718903070527 state 0x0 ref 1 00000100:00100000:2.0:1789760729.572152:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484199168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760729.572200:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20340736, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:2.0:1789760729.572209:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.572225:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:2.0:1789760729.572229:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760729.572236:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20340736->20344831] (req 20340736->20344831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.572250:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20340736->20344831] (req 20340736->20344831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760729.613145:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20340736 -> 20344832 00010000:00010000:1.0:1789760729.613167:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20340736->20344831] (req 20340736->20344831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.613181:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20340736->20344831] (req 20340736->20344831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.613198:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20340736->20344831] (req 20340736->20344831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.613206:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.613210:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20340736->20344831] (req 20340736->20344831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.613220:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20340736->20344831] (req 20340736->20344831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.613242:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20340736->20344831] (req 20340736->20344831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.613255:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208187 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20340736->20344831] (req 20340736->20344831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.613373:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484199168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41218us (41594us total) trans 4294972284 rc 0/0 00000100:00100000:1.0:1789760729.613384:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4995 00000100:00100000:1.0:1789760729.613393:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718867155498 time 718903070527 state 0x2 ref 0 00000001:00080000:0.0:1789760729.614157:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972284 is committed 00000100:00100000:3.0:1789760729.616986:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760729.619273:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484199296 00000100:00100000:0.0:1789760729.619287:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484199296 00000100:00100000:0.0:1789760729.619418:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718867155498 time 718903070527 state 0x0 ref 1 00000100:00100000:0.0:1789760729.619434:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718903330943 time 718903070527 state 0x1 ref 1 00000100:00100000:0.0:1789760729.619552:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718903330943 time 718950482181 state 0x0 ref 1 00000100:00100000:0.0:1789760729.619568:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760729.619615:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20344832, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:0.0:1789760729.619624:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.619640:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:0.0:1789760729.619644:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760729.619650:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20344832->20348927] (req 20344832->20348927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.619663:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20344832->20348927] (req 20344832->20348927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760729.656753:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972285 is committed 00002000:00010000:1.0:1789760729.656903:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20344832 -> 20348928 00010000:00010000:1.0:1789760729.656917:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20344832->20348927] (req 20344832->20348927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.656929:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20344832->20348927] (req 20344832->20348927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.656938:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20344832->20348927] (req 20344832->20348927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.656944:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.656949:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20344832->20348927] (req 20344832->20348927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.656959:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20344832->20348927] (req 20344832->20348927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.656983:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20344832->20348927] (req 20344832->20348927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.656991:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620818e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20344832->20348927] (req 20344832->20348927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.657121:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37553us (40139us total) trans 4294972285 rc 0/0 00000100:00100000:1.0:1789760729.657139:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4996 00000100:00100000:1.0:1789760729.657148:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718903330943 time 718950482181 state 0x2 ref 0 00000100:00100000:2.0:1789760729.661580:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760729.661708:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484199424 00000100:00100000:0.0:1789760729.661717:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484199424 00000100:00100000:0.0:1789760729.661776:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718903330943 time 718950482181 state 0x0 ref 1 00000100:00100000:0.0:1789760729.661792:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718950742597 time 718950482181 state 0x1 ref 1 00000100:00100000:0.0:1789760729.661858:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718950742597 time 718992788060 state 0x0 ref 1 00000100:00100000:0.0:1789760729.661872:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760729.661915:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20348928, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:0.0:1789760729.661923:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.661939:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=718s v=(0 0) 00010000:00010000:0.0:1789760729.661944:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760729.661950:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20348928->20353023] (req 20348928->20353023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.661964:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20348928->20353023] (req 20348928->20353023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760729.711914:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972286 is committed 00002000:00010000:2.0:1789760729.711977:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20348928 -> 20353024 00010000:00010000:2.0:1789760729.711991:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20348928->20353023] (req 20348928->20353023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.712003:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20348928->20353023] (req 20348928->20353023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.712012:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20348928->20353023] (req 20348928->20353023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.712019:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760729.712024:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20348928->20353023] (req 20348928->20353023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.712035:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20348928->20353023] (req 20348928->20353023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.712058:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20348928->20353023] (req 20348928->20353023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.712103:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208195 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20348928->20353023] (req 20348928->20353023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760729.712188:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50314us (50614us total) trans 4294972286 rc 0/0 00000100:00100000:2.0:1789760729.712198:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4997 00000100:00100000:2.0:1789760729.712208:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718950742597 time 718992788060 state 0x2 ref 0 00000100:00100000:2.0:1789760729.715608:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760729.715723:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484199552 00000100:00100000:1.0:1789760729.715733:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484199552 00000100:00100000:1.0:1789760729.715787:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718950742597 time 718992788060 state 0x0 ref 1 00000100:00100000:1.0:1789760729.715803:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718993048476 time 718992788060 state 0x1 ref 1 00000100:00100000:1.0:1789760729.720212:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718993048476 time 719051129351 state 0x0 ref 1 00000100:00100000:1.0:1789760729.720243:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760729.720305:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20353024, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:1.0:1789760729.720315:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.720331:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:1.0:1789760729.720336:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760729.720343:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20353024->20357119] (req 20353024->20357119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.720358:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20353024->20357119] (req 20353024->20357119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760729.749765:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972287 is committed 00002000:00010000:2.0:1789760729.749860:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20353024 -> 20357120 00010000:00010000:2.0:1789760729.749873:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20353024->20357119] (req 20353024->20357119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.749888:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20353024->20357119] (req 20353024->20357119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.749896:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20353024->20357119] (req 20353024->20357119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.749907:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760729.749913:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20353024->20357119] (req 20353024->20357119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.749922:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20353024->20357119] (req 20353024->20357119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.749941:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20353024->20357119] (req 20353024->20357119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.749948:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620819c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20353024->20357119] (req 20353024->20357119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760729.750106:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29872us (34503us total) trans 4294972287 rc 0/0 00000100:00100000:2.0:1789760729.750117:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4998 00000100:00100000:2.0:1789760729.750126:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718993048476 time 719051129351 state 0x2 ref 0 00000100:00100000:1.0:1789760729.754029:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760729.754272:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484199680 00000100:00100000:2.0:1789760729.754282:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484199680 00000100:00100000:2.0:1789760729.754330:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 718993048476 time 719051129351 state 0x0 ref 1 00000100:00100000:2.0:1789760729.754343:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719051389767 time 719051129351 state 0x1 ref 1 00000100:00100000:2.0:1789760729.754412:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719051389767 time 719085334021 state 0x0 ref 1 00000100:00100000:2.0:1789760729.754429:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760729.754471:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20357120, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:2.0:1789760729.754479:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.754501:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:2.0:1789760729.754506:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760729.754512:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20357120->20361215] (req 20357120->20361215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.754525:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20357120->20361215] (req 20357120->20361215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760729.782095:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972288 is committed 00002000:00010000:1.0:1789760729.782211:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20357120 -> 20361216 00010000:00010000:1.0:1789760729.782226:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20357120->20361215] (req 20357120->20361215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.782238:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20357120->20361215] (req 20357120->20361215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.782247:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20357120->20361215] (req 20357120->20361215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.782254:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.782259:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20357120->20361215] (req 20357120->20361215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.782269:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20357120->20361215] (req 20357120->20361215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.782288:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20357120->20361215] (req 20357120->20361215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.782300:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962081a3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20357120->20361215] (req 20357120->20361215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.782433:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28001us (28408us total) trans 4294972288 rc 0/0 00000100:00100000:1.0:1789760729.782443:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 4999 00000100:00100000:1.0:1789760729.782453:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719051389767 time 719085334021 state 0x2 ref 0 00000100:00100000:1.0:1789760729.785668:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760729.785777:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484199808 00000100:00100000:0.0:1789760729.785787:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484199808 00000100:00100000:0.0:1789760729.785847:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719051389767 time 719085334021 state 0x0 ref 1 00000100:00100000:0.0:1789760729.785861:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719085594437 time 719085334021 state 0x1 ref 1 00000100:00100000:0.0:1789760729.785893:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719085594437 time 719116824353 state 0x0 ref 1 00000100:00100000:0.0:1789760729.785906:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760729.785952:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20361216, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:0.0:1789760729.785960:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.785979:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:0.0:1789760729.785983:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760729.785989:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20361216->20365311] (req 20361216->20365311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760729.786003:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20361216->20365311] (req 20361216->20365311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760729.822680:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972289 is committed 00002000:00010000:1.0:1789760729.822824:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20361216 -> 20365312 00010000:00010000:1.0:1789760729.822839:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20361216->20365311] (req 20361216->20365311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.822851:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20361216->20365311] (req 20361216->20365311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.822859:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20361216->20365311] (req 20361216->20365311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.822865:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.822869:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20361216->20365311] (req 20361216->20365311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.822878:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20361216->20365311] (req 20361216->20365311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.822903:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20361216->20365311] (req 20361216->20365311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.822911:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962081aa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20361216->20365311] (req 20361216->20365311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.823040:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37132us (37375us total) trans 4294972289 rc 0/0 00000100:00100000:1.0:1789760729.823051:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5000 00000100:00100000:1.0:1789760729.823061:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719085594437 time 719116824353 state 0x2 ref 0 00000100:00100000:2.0:1789760729.826799:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760729.826880:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484199936 00000100:00100000:3.0:1789760729.826889:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484199936 00000100:00100000:3.0:1789760729.826938:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719085594437 time 719116824353 state 0x0 ref 1 00000100:00100000:3.0:1789760729.826953:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719117084769 time 719116824353 state 0x1 ref 1 00000100:00100000:3.0:1789760729.827009:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719117084769 time 719157939583 state 0x0 ref 1 00000100:00100000:3.0:1789760729.827021:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484199936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760729.827142:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20365312, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:3.0:1789760729.827151:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.827165:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:3.0:1789760729.827169:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760729.827175:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20365312->20369407] (req 20365312->20369407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.827196:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20365312->20369407] (req 20365312->20369407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760729.854619:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20365312 -> 20369408 00010000:00010000:3.0:1789760729.854640:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20365312->20369407] (req 20365312->20369407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.854654:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20365312->20369407] (req 20365312->20369407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.854665:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20365312->20369407] (req 20365312->20369407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.854672:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760729.854677:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20365312->20369407] (req 20365312->20369407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.854687:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20365312->20369407] (req 20365312->20369407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.854784:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20365312->20369407] (req 20365312->20369407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.854793:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081b1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20365312->20369407] (req 20365312->20369407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760729.854885:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484199936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27860us (28088us total) trans 4294972290 rc 0/0 00000100:00100000:3.0:1789760729.854896:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5001 00000100:00100000:3.0:1789760729.854906:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719117084769 time 719157939583 state 0x2 ref 0 00000001:00080000:1.0:1789760729.857253:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972290 is committed 00000100:00100000:1.0:1789760729.858998:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760729.859283:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484200064 00000100:00100000:2.0:1789760729.859295:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484200064 00000100:00100000:2.0:1789760729.859352:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719117084769 time 719157939583 state 0x0 ref 1 00000100:00100000:2.0:1789760729.859369:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719158199999 time 719157939583 state 0x1 ref 1 00000100:00100000:2.0:1789760729.859459:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719158199999 time 719190389174 state 0x0 ref 1 00000100:00100000:2.0:1789760729.859471:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484200064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760729.859534:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20369408, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:2.0:1789760729.859543:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.859560:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:2.0:1789760729.859564:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760729.859571:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20369408->20373503] (req 20369408->20373503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760729.859585:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20369408->20373503] (req 20369408->20373503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760729.891250:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972291 is committed 00002000:00010000:1.0:1789760729.891314:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20369408 -> 20373504 00010000:00010000:1.0:1789760729.891330:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20369408->20373503] (req 20369408->20373503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.891343:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20369408->20373503] (req 20369408->20373503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.891352:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20369408->20373503] (req 20369408->20373503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.891360:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.891365:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20369408->20373503] (req 20369408->20373503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.891375:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20369408->20373503] (req 20369408->20373503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.891395:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20369408->20373503] (req 20369408->20373503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.891412:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962081b8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20369408->20373503] (req 20369408->20373503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.891522:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484200064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32047us (32528us total) trans 4294972291 rc 0/0 00000100:00100000:1.0:1789760729.891534:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5002 00000100:00100000:1.0:1789760729.891546:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719158199999 time 719190389174 state 0x2 ref 0 00000100:00100000:2.0:1789760729.895282:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760729.895441:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484200192 00000100:00100000:1.0:1789760729.895452:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484200192 00000100:00100000:1.0:1789760729.895518:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719158199999 time 719190389174 state 0x0 ref 1 00000100:00100000:1.0:1789760729.895533:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719190649590 time 719190389174 state 0x1 ref 1 00000100:00100000:1.0:1789760729.895559:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719190649590 time 719226490702 state 0x0 ref 1 00000100:00100000:1.0:1789760729.895571:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484200192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760729.895616:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20373504, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:1.0:1789760729.895625:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.895642:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:1.0:1789760729.895646:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760729.895652:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20373504->20377599] (req 20373504->20377599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.895666:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20373504->20377599] (req 20373504->20377599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760729.927012:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972292 is committed 00002000:00010000:1.0:1789760729.927119:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20373504 -> 20377600 00010000:00010000:1.0:1789760729.927135:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20373504->20377599] (req 20373504->20377599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.927147:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20373504->20377599] (req 20373504->20377599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.927156:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20373504->20377599] (req 20373504->20377599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.927163:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.927168:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20373504->20377599] (req 20373504->20377599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.927177:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20373504->20377599] (req 20373504->20377599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.927212:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20373504->20377599] (req 20373504->20377599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.927220:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081bf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20373504->20377599] (req 20373504->20377599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.927312:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484200192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31736us (32033us total) trans 4294972292 rc 0/0 00000100:00100000:1.0:1789760729.927322:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5003 00000100:00100000:1.0:1789760729.927331:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719190649590 time 719226490702 state 0x2 ref 0 00000100:00100000:2.0:1789760729.930663:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760729.930934:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484200320 00000100:00100000:1.0:1789760729.930944:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484200320 00000100:00100000:1.0:1789760729.930999:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719190649590 time 719226490702 state 0x0 ref 1 00000100:00100000:1.0:1789760729.931015:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719226751118 time 719226490702 state 0x1 ref 1 00000100:00100000:1.0:1789760729.931038:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719226751118 time 719261968923 state 0x0 ref 1 00000100:00100000:1.0:1789760729.931050:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484200320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760729.931119:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20377600, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:1.0:1789760729.931130:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.931146:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:1.0:1789760729.931151:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760729.931157:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20377600->20381695] (req 20377600->20381695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.931169:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20377600->20381695] (req 20377600->20381695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760729.964978:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972293 is committed 00002000:00010000:3.0:1789760729.965165:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20377600 -> 20381696 00010000:00010000:3.0:1789760729.965181:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20377600->20381695] (req 20377600->20381695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.965257:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20377600->20381695] (req 20377600->20381695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.965265:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20377600->20381695] (req 20377600->20381695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.965272:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760729.965276:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20377600->20381695] (req 20377600->20381695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.965285:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20377600->20381695] (req 20377600->20381695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.965304:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20377600->20381695] (req 20377600->20381695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.965312:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962081c6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20377600->20381695] (req 20377600->20381695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760729.965464:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484200320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34411us (34804us total) trans 4294972293 rc 0/0 00000100:00100000:3.0:1789760729.965474:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5004 00000100:00100000:3.0:1789760729.965485:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719226751118 time 719261968923 state 0x2 ref 0 00000100:00100000:1.0:1789760729.969978:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760729.970054:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484200448 00000100:00100000:3.0:1789760729.970082:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484200448 00000100:00100000:3.0:1789760729.970132:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719226751118 time 719261968923 state 0x0 ref 1 00000100:00100000:3.0:1789760729.970147:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719262229339 time 719261968923 state 0x1 ref 1 00000100:00100000:3.0:1789760729.970257:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719262229339 time 719301187670 state 0x0 ref 1 00000100:00100000:3.0:1789760729.970270:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484200448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760729.970313:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20381696, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:3.0:1789760729.970322:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.970337:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:3.0:1789760729.970341:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760729.970347:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20381696->20385791] (req 20381696->20385791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.970359:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20381696->20385791] (req 20381696->20385791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760729.996248:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760729.996530:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972294 is committed 00002000:00010000:1.0:1789760729.996643:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20381696 -> 20385792 00010000:00010000:1.0:1789760729.996658:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20381696->20385791] (req 20381696->20385791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.996671:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20381696->20385791] (req 20381696->20385791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.996679:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20381696->20385791] (req 20381696->20385791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.996686:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760729.996690:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20381696->20385791] (req 20381696->20385791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.996766:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20381696->20385791] (req 20381696->20385791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.996784:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20381696->20385791] (req 20381696->20385791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760729.996796:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962081cd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20381696->20385791] (req 20381696->20385791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760729.996888:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484200448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26614us (26914us total) trans 4294972294 rc 0/0 00000100:00100000:1.0:1789760729.996899:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5005 00000100:00100000:1.0:1789760729.996906:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719262229339 time 719301187670 state 0x2 ref 0 00000100:00100000:0.0:1789760730.000468:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760730.000619:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484200576 00000100:00100000:2.0:1789760730.000629:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484200576 00000100:00100000:2.0:1789760730.000740:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719262229339 time 719301187670 state 0x0 ref 1 00000100:00100000:2.0:1789760730.000755:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719301448086 time 719301187670 state 0x1 ref 1 00000100:00100000:2.0:1789760730.000784:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719301448086 time 719331715143 state 0x0 ref 1 00000100:00100000:2.0:1789760730.000797:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484200576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760730.000849:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20385792, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:2.0:1789760730.000858:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.000872:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:2.0:1789760730.000877:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760730.000883:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20385792->20389887] (req 20385792->20389887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.000897:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20385792->20389887] (req 20385792->20389887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1789760730.001646:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:3.0:1789760730.025925:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972295 is committed 00002000:00010000:1.0:1789760730.026021:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20385792 -> 20389888 00010000:00010000:1.0:1789760730.026036:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20385792->20389887] (req 20385792->20389887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.026050:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20385792->20389887] (req 20385792->20389887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.026059:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20385792->20389887] (req 20385792->20389887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.026143:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760730.026148:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20385792->20389887] (req 20385792->20389887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.026158:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20385792->20389887] (req 20385792->20389887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.026263:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20385792->20389887] (req 20385792->20389887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.026273:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962081d4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20385792->20389887] (req 20385792->20389887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760730.026429:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484200576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25606us (25942us total) trans 4294972295 rc 0/0 00000100:00100000:1.0:1789760730.026439:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5006 00000100:00100000:1.0:1789760730.026450:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719301448086 time 719331715143 state 0x2 ref 0 00000100:00100000:2.0:1789760730.030622:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760730.030768:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484200704 00000100:00100000:1.0:1789760730.030778:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484200704 00000100:00100000:1.0:1789760730.030831:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719301448086 time 719331715143 state 0x0 ref 1 00000100:00100000:1.0:1789760730.030846:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719331975559 time 719331715143 state 0x1 ref 1 00000100:00100000:1.0:1789760730.030875:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719331975559 time 719361805997 state 0x0 ref 1 00000100:00100000:1.0:1789760730.030887:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484200704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760730.030937:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20389888, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:1.0:1789760730.030947:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.030963:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:1.0:1789760730.030968:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760730.030975:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20389888->20393983] (req 20389888->20393983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.030988:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20389888->20393983] (req 20389888->20393983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760730.063671:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972296 is committed 00002000:00010000:1.0:1789760730.063812:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20389888 -> 20393984 00010000:00010000:1.0:1789760730.063828:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20389888->20393983] (req 20389888->20393983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.063841:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20389888->20393983] (req 20389888->20393983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.063850:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20389888->20393983] (req 20389888->20393983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.063857:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760730.063862:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20389888->20393983] (req 20389888->20393983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.063871:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20389888->20393983] (req 20389888->20393983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.063890:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20389888->20393983] (req 20389888->20393983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.063898:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081db lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20389888->20393983] (req 20389888->20393983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760730.063987:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484200704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33098us (33372us total) trans 4294972296 rc 0/0 00000100:00100000:1.0:1789760730.063997:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5007 00000100:00100000:1.0:1789760730.064006:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719331975559 time 719361805997 state 0x2 ref 0 00000100:00100000:3.0:1789760730.069877:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760730.069974:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484200832 00000100:00100000:1.0:1789760730.069983:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484200832 00000100:00100000:1.0:1789760730.070046:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719331975559 time 719361805997 state 0x0 ref 1 00000100:00100000:1.0:1789760730.070062:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719362066413 time 719361805997 state 0x1 ref 1 00000100:00100000:1.0:1789760730.070123:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719362066413 time 719401053485 state 0x0 ref 1 00000100:00100000:1.0:1789760730.070137:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484200832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760730.070198:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20393984, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:1.0:1789760730.070208:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.070224:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:1.0:1789760730.070229:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760730.070235:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20393984->20398079] (req 20393984->20398079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.070249:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20393984->20398079] (req 20393984->20398079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760730.101986:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972297 is committed 00002000:00010000:0.0:1789760730.102682:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20393984 -> 20398080 00010000:00010000:0.0:1789760730.102699:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20393984->20398079] (req 20393984->20398079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.102714:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20393984->20398079] (req 20393984->20398079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.102723:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20393984->20398079] (req 20393984->20398079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.102731:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760730.102736:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20393984->20398079] (req 20393984->20398079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.102746:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20393984->20398079] (req 20393984->20398079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.102766:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20393984->20398079] (req 20393984->20398079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.102775:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962081e2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20393984->20398079] (req 20393984->20398079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760730.102909:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484200832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32769us (33034us total) trans 4294972297 rc 0/0 00000100:00100000:0.0:1789760730.102920:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5008 00000100:00100000:0.0:1789760730.102930:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719362066413 time 719401053485 state 0x2 ref 0 00000100:00100000:0.0:1789760730.107286:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760730.107464:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484200960 00000100:00100000:3.0:1789760730.107474:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484200960 00000100:00100000:3.0:1789760730.107532:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719362066413 time 719401053485 state 0x0 ref 1 00000100:00100000:3.0:1789760730.107548:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719401313901 time 719401053485 state 0x1 ref 1 00000100:00100000:3.0:1789760730.107569:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719401313901 time 719438499870 state 0x0 ref 1 00000100:00100000:3.0:1789760730.107585:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484200960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760730.107633:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20398080, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:3.0:1789760730.107642:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.107657:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:3.0:1789760730.107662:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760730.107668:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20398080->20402175] (req 20398080->20402175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.107683:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20398080->20402175] (req 20398080->20402175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760730.139978:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20398080 -> 20402176 00010000:00010000:3.0:1789760730.139998:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20398080->20402175] (req 20398080->20402175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.140011:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20398080->20402175] (req 20398080->20402175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.140023:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20398080->20402175] (req 20398080->20402175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.140030:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760730.140035:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20398080->20402175] (req 20398080->20402175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.140044:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20398080->20402175] (req 20398080->20402175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.140086:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20398080->20402175] (req 20398080->20402175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.140096:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962081e9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20398080->20402175] (req 20398080->20402175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760730.144706:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484200960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32603us (32905us total) trans 4294972298 rc 0/0 00000100:00100000:3.0:1789760730.144722:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5009 00000100:00100000:3.0:1789760730.144732:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719401313901 time 719438499870 state 0x2 ref 0 00000001:00080000:2.0:1789760730.144895:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972298 is committed 00000100:00100000:0.0:1789760730.145013:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760730.145242:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484201088 00000100:00100000:2.0:1789760730.145251:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484201088 00000100:00100000:2.0:1789760730.145461:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719401313901 time 719438499870 state 0x0 ref 1 00000100:00100000:2.0:1789760730.145475:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719438760286 time 719438499870 state 0x1 ref 1 00000100:00100000:2.0:1789760730.145543:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719438760286 time 719476473335 state 0x0 ref 1 00000100:00100000:2.0:1789760730.145557:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484201088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760730.145605:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20402176, mtime: 0x6aad94d9, atime: 0x0, ctime: 0x6aad94d9, blocks: 0xa007 00010000:00010000:2.0:1789760730.145613:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.145627:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:2.0:1789760730.145632:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760730.145638:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20402176->20406271] (req 20402176->20406271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.145658:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20402176->20406271] (req 20402176->20406271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760730.202893:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972299 is committed 00002000:00010000:2.0:1789760730.203109:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20402176 -> 20406272 00002000:00010000:2.0:1789760730.203115:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760729 -> 1789760730 00002000:00010000:2.0:1789760730.203118:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760729 -> 1789760730 00010000:00010000:2.0:1789760730.203133:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20402176->20406271] (req 20402176->20406271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.203147:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20402176->20406271] (req 20402176->20406271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.203157:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20402176->20406271] (req 20402176->20406271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.203165:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760730.203171:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20402176->20406271] (req 20402176->20406271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.203252:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20402176->20406271] (req 20402176->20406271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.203279:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20402176->20406271] (req 20402176->20406271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.203293:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962081f0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20402176->20406271] (req 20402176->20406271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760730.203423:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484201088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57862us (58404us total) trans 4294972299 rc 0/0 00000100:00100000:2.0:1789760730.203435:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5010 00000100:00100000:2.0:1789760730.203519:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719438760286 time 719476473335 state 0x2 ref 0 00000100:00100000:0.0:1789760730.208059:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760730.208252:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484201216 00000100:00100000:1.0:1789760730.208265:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484201216 00000100:00100000:1.0:1789760730.208340:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719438760286 time 719476473335 state 0x0 ref 1 00000100:00100000:1.0:1789760730.208360:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719476733751 time 719476473335 state 0x1 ref 1 00000100:00100000:1.0:1789760730.208411:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719476733751 time 719539332311 state 0x0 ref 1 00000100:00100000:1.0:1789760730.208425:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484201216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760730.208502:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20406272, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:1.0:1789760730.208513:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.208532:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:1.0:1789760730.208537:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760730.208545:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20406272->20410367] (req 20406272->20410367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.208562:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20406272->20410367] (req 20406272->20410367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760730.262012:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20406272 -> 20410368 00010000:00010000:1.0:1789760730.262039:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20406272->20410367] (req 20406272->20410367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.262061:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20406272->20410367] (req 20406272->20410367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.262108:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20406272->20410367] (req 20406272->20410367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.262117:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760730.262123:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20406272->20410367] (req 20406272->20410367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.262134:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20406272->20410367] (req 20406272->20410367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.262159:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20406272->20410367] (req 20406272->20410367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.262168:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962081f7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20406272->20410367] (req 20406272->20410367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760730.262309:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484201216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53879us (54255us total) trans 4294972300 rc 0/0 00000100:00100000:1.0:1789760730.262322:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5011 00000100:00100000:1.0:1789760730.262336:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719476733751 time 719539332311 state 0x2 ref 0 00000001:00080000:3.0:1789760730.263895:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972300 is committed 00000100:00100000:1.0:1789760730.266615:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760730.266826:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484201344 00000100:00100000:0.0:1789760730.266840:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484201344 00000100:00100000:0.0:1789760730.266923:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719476733751 time 719539332311 state 0x0 ref 1 00000100:00100000:0.0:1789760730.266942:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719539592727 time 719539332311 state 0x1 ref 1 00000100:00100000:0.0:1789760730.266979:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719539592727 time 719597909368 state 0x0 ref 1 00000100:00100000:0.0:1789760730.266995:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484201344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760730.267059:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20410368, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:0.0:1789760730.267102:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.267125:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:0.0:1789760730.267130:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760730.267137:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20410368->20414463] (req 20410368->20414463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.267155:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20410368->20414463] (req 20410368->20414463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760730.299613:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972301 is committed 00002000:00010000:2.0:1789760730.299833:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20410368 -> 20414464 00010000:00010000:2.0:1789760730.299853:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20410368->20414463] (req 20410368->20414463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.299867:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20410368->20414463] (req 20410368->20414463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.299877:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20410368->20414463] (req 20410368->20414463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.299884:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760730.299889:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20410368->20414463] (req 20410368->20414463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.299899:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20410368->20414463] (req 20410368->20414463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.299928:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20410368->20414463] (req 20410368->20414463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.299936:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962081fe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20410368->20414463] (req 20410368->20414463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760730.300058:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484201344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33061us (33451us total) trans 4294972301 rc 0/0 00000100:00100000:2.0:1789760730.300115:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5012 00000100:00100000:2.0:1789760730.300127:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719539592727 time 719597909368 state 0x2 ref 0 00000100:00100000:2.0:1789760730.303980:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760730.304273:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484201472 00000100:00100000:3.0:1789760730.304288:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484201472 00000100:00100000:3.0:1789760730.304366:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719539592727 time 719597909368 state 0x0 ref 1 00000100:00100000:3.0:1789760730.304387:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719598169784 time 719597909368 state 0x1 ref 1 00000100:00100000:3.0:1789760730.304427:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719598169784 time 719635357310 state 0x0 ref 1 00000100:00100000:3.0:1789760730.304441:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484201472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760730.304517:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20414464, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:3.0:1789760730.304528:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.304548:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:3.0:1789760730.304553:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760730.304562:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20414464->20418559] (req 20414464->20418559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.304648:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20414464->20418559] (req 20414464->20418559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760730.347111:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972302 is committed 00002000:00010000:0.0:1789760730.347200:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20414464 -> 20418560 00010000:00010000:0.0:1789760730.347219:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20414464->20418559] (req 20414464->20418559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.347235:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20414464->20418559] (req 20414464->20418559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.347244:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20414464->20418559] (req 20414464->20418559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.347253:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760730.347259:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20414464->20418559] (req 20414464->20418559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.347270:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20414464->20418559] (req 20414464->20418559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.347302:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20414464->20418559] (req 20414464->20418559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.347312:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208205 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20414464->20418559] (req 20414464->20418559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760730.347555:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484201472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43108us (43580us total) trans 4294972302 rc 0/0 00000100:00100000:0.0:1789760730.347567:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5013 00000100:00100000:0.0:1789760730.347578:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719598169784 time 719635357310 state 0x2 ref 0 00000100:00100000:0.0:1789760730.355930:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760730.356183:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484201600 00000100:00100000:2.0:1789760730.356271:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484201600 00000100:00100000:2.0:1789760730.356347:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719598169784 time 719635357310 state 0x0 ref 1 00000100:00100000:2.0:1789760730.356368:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719635617726 time 719635357310 state 0x1 ref 1 00000100:00100000:2.0:1789760730.356527:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719635617726 time 719687455416 state 0x0 ref 1 00000100:00100000:2.0:1789760730.356545:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484201600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760730.356615:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20418560, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:2.0:1789760730.356627:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.356656:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:2.0:1789760730.356662:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760730.356802:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20418560->20422655] (req 20418560->20422655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.356824:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20418560->20422655] (req 20418560->20422655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760730.394760:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20418560 -> 20422656 00010000:00010000:3.0:1789760730.394787:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20418560->20422655] (req 20418560->20422655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.394802:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20418560->20422655] (req 20418560->20422655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.394812:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20418560->20422655] (req 20418560->20422655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.394826:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760730.394832:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20418560->20422655] (req 20418560->20422655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.394847:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20418560->20422655] (req 20418560->20422655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.394881:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20418560->20422655] (req 20418560->20422655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.394890:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620820c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20418560->20422655] (req 20418560->20422655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760730.395018:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484201600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38470us (39096us total) trans 4294972303 rc 0/0 00000100:00100000:3.0:1789760730.395030:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5014 00000100:00100000:3.0:1789760730.395050:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719635617726 time 719687455416 state 0x2 ref 0 00000001:00080000:0.0:1789760730.395782:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972303 is committed 00000100:00100000:1.0:1789760730.411912:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760730.412138:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484201728 00000100:00100000:2.0:1789760730.412152:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484201728 00000100:00100000:2.0:1789760730.412235:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719635617726 time 719687455416 state 0x0 ref 1 00000100:00100000:2.0:1789760730.412254:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719687715832 time 719687455416 state 0x1 ref 1 00000100:00100000:2.0:1789760730.412487:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719687715832 time 719743415909 state 0x0 ref 1 00000100:00100000:2.0:1789760730.412503:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484201728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760730.412568:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20422656, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:2.0:1789760730.412579:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.412598:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:2.0:1789760730.412603:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760730.412611:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20422656->20426751] (req 20422656->20426751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.412633:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20422656->20426751] (req 20422656->20426751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760730.461049:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972304 is committed 00002000:00010000:2.0:1789760730.461321:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20422656 -> 20426752 00010000:00010000:2.0:1789760730.461342:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20422656->20426751] (req 20422656->20426751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.461358:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20422656->20426751] (req 20422656->20426751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.461369:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20422656->20426751] (req 20422656->20426751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.461377:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760730.461392:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20422656->20426751] (req 20422656->20426751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.461410:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20422656->20426751] (req 20422656->20426751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.461435:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20422656->20426751] (req 20422656->20426751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.461444:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208213 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20422656->20426751] (req 20422656->20426751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760730.462013:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484201728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49504us (51939us total) trans 4294972304 rc 0/0 00000100:00100000:2.0:1789760730.462027:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5015 00000100:00100000:2.0:1789760730.462040:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719687715832 time 719743415909 state 0x2 ref 0 00000100:00100000:2.0:1789760730.468525:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760730.468721:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484201856 00000100:00100000:0.0:1789760730.468734:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484201856 00000100:00100000:0.0:1789760730.468812:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719687715832 time 719743415909 state 0x0 ref 1 00000100:00100000:0.0:1789760730.468831:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719743676325 time 719743415909 state 0x1 ref 1 00000100:00100000:0.0:1789760730.468908:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719743676325 time 719799838858 state 0x0 ref 1 00000100:00100000:0.0:1789760730.468926:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484201856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760730.469015:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20426752, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:0.0:1789760730.469026:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.469058:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:0.0:1789760730.469119:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760730.469130:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20426752->20430847] (req 20426752->20430847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.469150:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20426752->20430847] (req 20426752->20430847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760730.521512:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972305 is committed 00002000:00010000:0.0:1789760730.521749:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20426752 -> 20430848 00010000:00010000:0.0:1789760730.521770:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20426752->20430847] (req 20426752->20430847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.521785:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20426752->20430847] (req 20426752->20430847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.521795:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20426752->20430847] (req 20426752->20430847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.521803:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760730.521809:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20426752->20430847] (req 20426752->20430847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.521819:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20426752->20430847] (req 20426752->20430847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.521843:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20426752->20430847] (req 20426752->20430847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.521852:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620821a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20426752->20430847] (req 20426752->20430847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760730.521978:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484201856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53051us (53462us total) trans 4294972305 rc 0/0 00000100:00100000:0.0:1789760730.521991:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5016 00000100:00100000:0.0:1789760730.522003:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719743676325 time 719799838858 state 0x2 ref 0 00000100:00100000:3.0:1789760730.524855:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760730.524976:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484201984 00000100:00100000:1.0:1789760730.524990:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484201984 00000100:00100000:1.0:1789760730.525102:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719743676325 time 719799838858 state 0x0 ref 1 00000100:00100000:1.0:1789760730.525121:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719800099274 time 719799838858 state 0x1 ref 1 00000100:00100000:1.0:1789760730.525155:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719800099274 time 719856084976 state 0x0 ref 1 00000100:00100000:1.0:1789760730.525169:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484201984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760730.525233:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20430848, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:1.0:1789760730.525244:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.525263:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:1.0:1789760730.525268:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760730.525275:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20430848->20434943] (req 20430848->20434943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.525291:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20430848->20434943] (req 20430848->20434943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760730.550114:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972306 is committed 00002000:00010000:2.0:1789760730.550646:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20430848 -> 20434944 00010000:00010000:2.0:1789760730.550670:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20430848->20434943] (req 20430848->20434943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.550686:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20430848->20434943] (req 20430848->20434943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.550697:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20430848->20434943] (req 20430848->20434943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.550706:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760730.550712:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20430848->20434943] (req 20430848->20434943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.550723:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20430848->20434943] (req 20430848->20434943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.550749:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20430848->20434943] (req 20430848->20434943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.550763:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208221 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20430848->20434943] (req 20430848->20434943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760730.550890:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484201984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25717us (26044us total) trans 4294972306 rc 0/0 00000100:00100000:2.0:1789760730.550903:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5017 00000100:00100000:2.0:1789760730.550915:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719800099274 time 719856084976 state 0x2 ref 0 00000100:00100000:1.0:1789760730.554518:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760730.554631:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484202112 00000100:00100000:3.0:1789760730.554645:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484202112 00000100:00100000:3.0:1789760730.554718:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719800099274 time 719856084976 state 0x0 ref 1 00000100:00100000:3.0:1789760730.554737:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719856345392 time 719856084976 state 0x1 ref 1 00000100:00100000:3.0:1789760730.554777:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719856345392 time 719885707156 state 0x0 ref 1 00000100:00100000:3.0:1789760730.554791:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484202112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760730.554854:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20434944, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:3.0:1789760730.554864:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.554884:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:3.0:1789760730.554890:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760730.554897:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20434944->20439039] (req 20434944->20439039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.554914:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20434944->20439039] (req 20434944->20439039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760730.586825:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20434944 -> 20439040 00010000:00010000:1.0:1789760730.586850:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20434944->20439039] (req 20434944->20439039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.586872:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20434944->20439039] (req 20434944->20439039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.586882:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20434944->20439039] (req 20434944->20439039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.586891:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760730.586897:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20434944->20439039] (req 20434944->20439039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.586907:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20434944->20439039] (req 20434944->20439039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.586933:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20434944->20439039] (req 20434944->20439039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.586942:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208228 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20434944->20439039] (req 20434944->20439039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760730.587105:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484202112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32269us (32554us total) trans 4294972307 rc 0/0 00000100:00100000:1.0:1789760730.587119:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5018 00000100:00100000:1.0:1789760730.587131:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719856345392 time 719885707156 state 0x2 ref 0 00000001:00080000:2.0:1789760730.587288:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972307 is committed 00000100:00100000:0.0:1789760730.590330:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760730.590436:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484202240 00000100:00100000:2.0:1789760730.590449:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484202240 00000100:00100000:2.0:1789760730.590534:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719856345392 time 719885707156 state 0x0 ref 1 00000100:00100000:2.0:1789760730.590553:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719885967572 time 719885707156 state 0x1 ref 1 00000100:00100000:2.0:1789760730.590586:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719885967572 time 719921516481 state 0x0 ref 1 00000100:00100000:2.0:1789760730.590601:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484202240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760730.590670:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20439040, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:2.0:1789760730.590681:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.590701:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:2.0:1789760730.590707:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760730.590714:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20439040->20443135] (req 20439040->20443135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.590730:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20439040->20443135] (req 20439040->20443135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760730.649547:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972308 is committed 00002000:00010000:3.0:1789760730.649686:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20439040 -> 20443136 00010000:00010000:3.0:1789760730.649706:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20439040->20443135] (req 20439040->20443135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.649721:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20439040->20443135] (req 20439040->20443135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.649731:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20439040->20443135] (req 20439040->20443135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.649739:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760730.649746:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20439040->20443135] (req 20439040->20443135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.649756:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20439040->20443135] (req 20439040->20443135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.649782:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20439040->20443135] (req 20439040->20443135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.649791:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620822f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20439040->20443135] (req 20439040->20443135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760730.649912:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484202240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59306us (59590us total) trans 4294972308 rc 0/0 00000100:00100000:3.0:1789760730.649925:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5019 00000100:00100000:3.0:1789760730.649937:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719885967572 time 719921516481 state 0x2 ref 0 00000100:00100000:3.0:1789760730.652310:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760730.652427:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484202368 00000100:00100000:0.0:1789760730.652441:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484202368 00000100:00100000:0.0:1789760730.652533:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719885967572 time 719921516481 state 0x0 ref 1 00000100:00100000:0.0:1789760730.652553:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719921776897 time 719921516481 state 0x1 ref 1 00000100:00100000:0.0:1789760730.652592:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719921776897 time 719983522084 state 0x0 ref 1 00000100:00100000:0.0:1789760730.652608:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484202368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760730.652673:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20443136, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:0.0:1789760730.652683:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.652704:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=719s v=(0 0) 00010000:00010000:0.0:1789760730.652709:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760730.652717:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20443136->20447231] (req 20443136->20447231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.652733:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20443136->20447231] (req 20443136->20447231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760730.690768:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972309 is committed 00002000:00010000:1.0:1789760730.691565:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20443136 -> 20447232 00010000:00010000:1.0:1789760730.691585:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20443136->20447231] (req 20443136->20447231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.691601:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20443136->20447231] (req 20443136->20447231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.691613:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20443136->20447231] (req 20443136->20447231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.691622:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760730.691628:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20443136->20447231] (req 20443136->20447231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.691638:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20443136->20447231] (req 20443136->20447231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.691666:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20443136->20447231] (req 20443136->20447231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.691675:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208236 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20443136->20447231] (req 20443136->20447231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760730.691847:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484202368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39235us (39542us total) trans 4294972309 rc 0/0 00000100:00100000:1.0:1789760730.691859:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5020 00000100:00100000:1.0:1789760730.691872:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719921776897 time 719983522084 state 0x2 ref 0 00000100:00100000:1.0:1789760730.696390:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760730.696466:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484202496 00000100:00100000:2.0:1789760730.696480:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484202496 00000100:00100000:2.0:1789760730.696555:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719921776897 time 719983522084 state 0x0 ref 1 00000100:00100000:2.0:1789760730.696574:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719983782500 time 719983522084 state 0x1 ref 1 00000100:00100000:2.0:1789760730.696615:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719983782500 time 720027541243 state 0x0 ref 1 00000100:00100000:2.0:1789760730.696631:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484202496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760730.696709:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20447232, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:2.0:1789760730.696719:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.696738:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:2.0:1789760730.696743:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760730.696751:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20447232->20451327] (req 20447232->20451327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.696766:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20447232->20451327] (req 20447232->20451327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760730.734666:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972310 is committed 00002000:00010000:2.0:1789760730.734926:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20447232 -> 20451328 00010000:00010000:2.0:1789760730.734947:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20447232->20451327] (req 20447232->20451327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.734963:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20447232->20451327] (req 20447232->20451327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.734973:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20447232->20451327] (req 20447232->20451327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.734982:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760730.734989:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20447232->20451327] (req 20447232->20451327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.735001:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20447232->20451327] (req 20447232->20451327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.735026:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20447232->20451327] (req 20447232->20451327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.735036:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620823d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20447232->20451327] (req 20447232->20451327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760730.735201:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484202496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38568us (38820us total) trans 4294972310 rc 0/0 00000100:00100000:2.0:1789760730.735218:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5021 00000100:00100000:2.0:1789760730.735231:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719983782500 time 720027541243 state 0x2 ref 0 00000100:00100000:0.0:1789760730.738806:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760730.738894:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484202624 00000100:00100000:3.0:1789760730.738908:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484202624 00000100:00100000:3.0:1789760730.738980:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 719983782500 time 720027541243 state 0x0 ref 1 00000100:00100000:3.0:1789760730.738999:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720027801659 time 720027541243 state 0x1 ref 1 00000100:00100000:3.0:1789760730.739032:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720027801659 time 720069962650 state 0x0 ref 1 00000100:00100000:3.0:1789760730.739046:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484202624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760730.739135:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20451328, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:3.0:1789760730.739147:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.739167:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:3.0:1789760730.739173:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760730.739455:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20451328->20455423] (req 20451328->20455423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.739478:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20451328->20455423] (req 20451328->20455423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760730.777630:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20451328 -> 20455424 00010000:00010000:3.0:1789760730.777656:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20451328->20455423] (req 20451328->20455423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.777672:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20451328->20455423] (req 20451328->20455423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.777683:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20451328->20455423] (req 20451328->20455423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.777692:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760730.777698:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20451328->20455423] (req 20451328->20455423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.777709:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20451328->20455423] (req 20451328->20455423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.777737:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20451328->20455423] (req 20451328->20455423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.777746:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208244 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20451328->20455423] (req 20451328->20455423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760730.777872:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484202624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38820us (39073us total) trans 4294972311 rc 0/0 00000100:00100000:3.0:1789760730.777885:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5022 00000100:00100000:3.0:1789760730.777908:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720027801659 time 720069962650 state 0x2 ref 0 00000001:00080000:1.0:1789760730.777974:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972311 is committed 00000100:00100000:3.0:1789760730.781000:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760730.781112:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484202752 00000100:00100000:2.0:1789760730.781126:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484202752 00000100:00100000:2.0:1789760730.783047:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720027801659 time 720069962650 state 0x0 ref 1 00000100:00100000:2.0:1789760730.783086:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720070223066 time 720069962650 state 0x1 ref 1 00000100:00100000:2.0:1789760730.783142:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720070223066 time 720114072836 state 0x0 ref 1 00000100:00100000:2.0:1789760730.783170:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484202752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760730.783242:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20455424, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:2.0:1789760730.783253:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.783272:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:2.0:1789760730.783276:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760730.783284:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20455424->20459519] (req 20455424->20459519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760730.783300:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20455424->20459519] (req 20455424->20459519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760730.836427:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972312 is committed 00002000:00010000:3.0:1789760730.836653:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20455424 -> 20459520 00010000:00010000:3.0:1789760730.836673:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20455424->20459519] (req 20455424->20459519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.836686:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20455424->20459519] (req 20455424->20459519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.836697:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20455424->20459519] (req 20455424->20459519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.836705:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760730.836710:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20455424->20459519] (req 20455424->20459519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.836726:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20455424->20459519] (req 20455424->20459519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.836751:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20455424->20459519] (req 20455424->20459519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760730.836760:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620824b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20455424->20459519] (req 20455424->20459519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760730.836885:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484202752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53722us (55890us total) trans 4294972312 rc 0/0 00000100:00100000:3.0:1789760730.836902:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5023 00000100:00100000:3.0:1789760730.836918:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720070223066 time 720114072836 state 0x2 ref 0 00000100:00100000:0.0:1789760730.840021:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760730.840335:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484202880 00000100:00100000:1.0:1789760730.840348:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484202880 00000100:00100000:1.0:1789760730.840496:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720070223066 time 720114072836 state 0x0 ref 1 00000100:00100000:1.0:1789760730.840516:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720114333252 time 720114072836 state 0x1 ref 1 00000100:00100000:1.0:1789760730.840588:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720114333252 time 720171517331 state 0x0 ref 1 00000100:00100000:1.0:1789760730.840604:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484202880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760730.840671:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20459520, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:1.0:1789760730.840683:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.840703:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:1.0:1789760730.840709:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760730.840717:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20459520->20463615] (req 20459520->20463615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.840734:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20459520->20463615] (req 20459520->20463615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760730.886599:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972313 is committed 00002000:00010000:0.0:1789760730.886746:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20459520 -> 20463616 00010000:00010000:0.0:1789760730.886766:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20459520->20463615] (req 20459520->20463615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.886780:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20459520->20463615] (req 20459520->20463615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.886790:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20459520->20463615] (req 20459520->20463615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.886797:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760730.886802:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20459520->20463615] (req 20459520->20463615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.886812:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20459520->20463615] (req 20459520->20463615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.886838:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20459520->20463615] (req 20459520->20463615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.886852:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208252 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20459520->20463615] (req 20459520->20463615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760730.887360:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484202880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46751us (47343us total) trans 4294972313 rc 0/0 00000100:00100000:0.0:1789760730.887373:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5024 00000100:00100000:0.0:1789760730.887386:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720114333252 time 720171517331 state 0x2 ref 0 00000100:00100000:0.0:1789760730.899595:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760730.899711:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484203008 00000100:00100000:1.0:1789760730.899725:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484203008 00000100:00100000:1.0:1789760730.899799:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720114333252 time 720171517331 state 0x0 ref 1 00000100:00100000:1.0:1789760730.899820:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720171777747 time 720171517331 state 0x1 ref 1 00000100:00100000:1.0:1789760730.899854:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720171777747 time 720230784057 state 0x0 ref 1 00000100:00100000:1.0:1789760730.899868:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484203008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760730.899936:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20463616, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:1.0:1789760730.899947:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.899967:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:1.0:1789760730.899973:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760730.899981:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20463616->20467711] (req 20463616->20467711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.899997:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20463616->20467711] (req 20463616->20467711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760730.931143:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972314 is committed 00002000:00010000:1.0:1789760730.931228:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20463616 -> 20467712 00010000:00010000:1.0:1789760730.931248:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20463616->20467711] (req 20463616->20467711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.931263:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20463616->20467711] (req 20463616->20467711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.931273:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20463616->20467711] (req 20463616->20467711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.931286:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760730.931292:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20463616->20467711] (req 20463616->20467711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.931303:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20463616->20467711] (req 20463616->20467711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.931327:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20463616->20467711] (req 20463616->20467711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760730.931336:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208259 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20463616->20467711] (req 20463616->20467711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760730.931636:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484203008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31764us (32049us total) trans 4294972314 rc 0/0 00000100:00100000:1.0:1789760730.931648:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5025 00000100:00100000:1.0:1789760730.931659:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720171777747 time 720230784057 state 0x2 ref 0 00000100:00100000:3.0:1789760730.935753:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760730.936291:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484203136 00000100:00100000:1.0:1789760730.936306:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484203136 00000100:00100000:0.0:1789760730.936581:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720171777747 time 720230784057 state 0x0 ref 1 00000100:00100000:0.0:1789760730.936600:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720231044473 time 720230784057 state 0x1 ref 1 00000100:00100000:0.0:1789760730.936635:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720231044473 time 720267565277 state 0x0 ref 1 00000100:00100000:0.0:1789760730.936650:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484203136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760730.936717:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20467712, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:0.0:1789760730.936728:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.936749:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:0.0:1789760730.936754:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760730.936762:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20467712->20471807] (req 20467712->20471807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.936854:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20467712->20471807] (req 20467712->20471807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760730.993638:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20467712 -> 20471808 00010000:00010000:0.0:1789760730.993667:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20467712->20471807] (req 20467712->20471807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.993683:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20467712->20471807] (req 20467712->20471807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.993693:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20467712->20471807] (req 20467712->20471807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.993701:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760730.993707:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20467712->20471807] (req 20467712->20471807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.993716:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20467712->20471807] (req 20467712->20471807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.993741:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20467712->20471807] (req 20467712->20471807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760730.993748:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208260 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20467712->20471807] (req 20467712->20471807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760730.994771:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484203136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58112us (59021us total) trans 4294972315 rc 0/0 00000100:00100000:0.0:1789760730.994787:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5026 00000100:00100000:0.0:1789760730.994800:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720231044473 time 720267565277 state 0x2 ref 0 00000001:00080000:0.0:1789760730.994861:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972315 is committed 00000100:00100000:0.0:1789760731.003972:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760731.006492:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484203264 00000100:00100000:3.0:1789760731.006589:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484203264 00000100:00100000:2.0:1789760731.009310:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720231044473 time 720267565277 state 0x0 ref 1 00000100:00100000:2.0:1789760731.009339:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720267825693 time 720267565277 state 0x1 ref 1 00000100:00100000:2.0:1789760731.009419:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720267825693 time 720340342693 state 0x0 ref 1 00000100:00100000:2.0:1789760731.009439:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484203264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760731.009507:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20471808, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:2.0:1789760731.009518:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.009538:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:2.0:1789760731.009544:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760731.009551:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20471808->20475903] (req 20471808->20475903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.009568:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20471808->20475903] (req 20471808->20475903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.030499:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760731.073594:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972316 is committed 00002000:00010000:3.0:1789760731.074018:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20471808 -> 20475904 00010000:00010000:3.0:1789760731.074043:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20471808->20475903] (req 20471808->20475903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.074057:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20471808->20475903] (req 20471808->20475903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.074095:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20471808->20475903] (req 20471808->20475903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.074104:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760731.074110:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20471808->20475903] (req 20471808->20475903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.074121:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20471808->20475903] (req 20471808->20475903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.074145:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20471808->20475903] (req 20471808->20475903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.074152:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208267 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20471808->20475903] (req 20471808->20475903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760731.074301:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484203264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 64861us (70336us total) trans 4294972316 rc 0/0 00000100:00100000:3.0:1789760731.074313:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5027 00000100:00100000:3.0:1789760731.074330:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720267825693 time 720340342693 state 0x2 ref 0 00000100:00100000:2.0:1789760731.077956:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760731.080511:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484203392 00000100:00100000:1.0:1789760731.080528:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484203392 00000100:00100000:1.0:1789760731.080593:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720267825693 time 720340342693 state 0x0 ref 1 00000100:00100000:1.0:1789760731.080608:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720340603109 time 720340342693 state 0x1 ref 1 00000100:00100000:1.0:1789760731.080643:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720340603109 time 720411573755 state 0x0 ref 1 00000100:00100000:1.0:1789760731.080657:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484203392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760731.080703:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20475904, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:1.0:1789760731.080711:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.080725:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:1.0:1789760731.080730:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760731.080736:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20475904->20479999] (req 20475904->20479999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.080751:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20475904->20479999] (req 20475904->20479999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760731.146403:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972317 is committed 00002000:00010000:2.0:1789760731.146875:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20475904 -> 20480000 00010000:00010000:2.0:1789760731.146896:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20475904->20479999] (req 20475904->20479999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.146912:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20475904->20479999] (req 20475904->20479999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.146929:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20475904->20479999] (req 20475904->20479999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.146939:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.146945:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20475904->20479999] (req 20475904->20479999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.146956:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20475904->20479999] (req 20475904->20479999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.146981:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20475904->20479999] (req 20475904->20479999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.146988:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620826e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20475904->20479999] (req 20475904->20479999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.147157:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484203392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66496us (69210us total) trans 4294972317 rc 0/0 00000100:00100000:2.0:1789760731.147169:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5028 00000100:00100000:2.0:1789760731.147191:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720340603109 time 720411573755 state 0x2 ref 0 00000100:00100000:0.0:1789760731.150706:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760731.150959:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484203520 00000100:00100000:1.0:1789760731.150972:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484203520 00000100:00100000:1.0:1789760731.151060:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720340603109 time 720411573755 state 0x0 ref 1 00000100:00100000:1.0:1789760731.151120:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720411834171 time 720411573755 state 0x1 ref 1 00000100:00100000:1.0:1789760731.151153:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720411834171 time 720482083619 state 0x0 ref 1 00000100:00100000:1.0:1789760731.151167:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484203520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760731.151235:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20480000, mtime: 0x6aad94da, atime: 0x0, ctime: 0x6aad94da, blocks: 0xa007 00010000:00010000:1.0:1789760731.151247:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.151268:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:1.0:1789760731.151274:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760731.151282:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20480000->20484095] (req 20480000->20484095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.151299:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20480000->20484095] (req 20480000->20484095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760731.257321:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972318 is committed 00002000:00010000:2.0:1789760731.257641:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20480000 -> 20484096 00002000:00010000:2.0:1789760731.257648:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760730 -> 1789760731 00002000:00010000:2.0:1789760731.257651:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760730 -> 1789760731 00010000:00010000:2.0:1789760731.257669:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20480000->20484095] (req 20480000->20484095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.257686:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20480000->20484095] (req 20480000->20484095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.257696:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20480000->20484095] (req 20480000->20484095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.257705:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.257711:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20480000->20484095] (req 20480000->20484095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.257731:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20480000->20484095] (req 20480000->20484095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.257759:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20480000->20484095] (req 20480000->20484095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.257768:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208275 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20480000->20484095] (req 20480000->20484095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.257893:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484203520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 106721us (107194us total) trans 4294972318 rc 0/0 00000100:00100000:2.0:1789760731.257905:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5029 00000100:00100000:2.0:1789760731.257918:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720411834171 time 720482083619 state 0x2 ref 0 00000100:00100000:0.0:1789760731.260825:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760731.260951:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484203648 00000100:00100000:3.0:1789760731.261020:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484203648 00000100:00100000:3.0:1789760731.261129:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720411834171 time 720482083619 state 0x0 ref 1 00000100:00100000:3.0:1789760731.261148:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720482344035 time 720482083619 state 0x1 ref 1 00000100:00100000:3.0:1789760731.261177:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720482344035 time 720592107748 state 0x0 ref 1 00000100:00100000:3.0:1789760731.261196:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484203648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760731.262057:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20484096, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:3.0:1789760731.262243:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.262330:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:3.0:1789760731.262336:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760731.262344:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20484096->20488191] (req 20484096->20488191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.262364:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20484096->20488191] (req 20484096->20488191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760731.320109:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972319 is committed 00002000:00010000:3.0:1789760731.320328:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20484096 -> 20488192 00010000:00010000:3.0:1789760731.320349:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20484096->20488191] (req 20484096->20488191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.320364:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20484096->20488191] (req 20484096->20488191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.320373:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20484096->20488191] (req 20484096->20488191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.320381:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760731.320387:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20484096->20488191] (req 20484096->20488191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.320409:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20484096->20488191] (req 20484096->20488191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.320435:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20484096->20488191] (req 20484096->20488191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.320453:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620827c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20484096->20488191] (req 20484096->20488191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760731.320703:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484203648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59503us (59887us total) trans 4294972319 rc 0/0 00000100:00100000:3.0:1789760731.320716:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5030 00000100:00100000:3.0:1789760731.320730:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720482344035 time 720592107748 state 0x2 ref 0 00000100:00100000:0.0:1789760731.324413:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760731.324568:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484203776 00000100:00100000:3.0:1789760731.324582:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484203776 00000100:00100000:3.0:1789760731.324662:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720482344035 time 720592107748 state 0x0 ref 1 00000100:00100000:3.0:1789760731.324681:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720592368164 time 720592107748 state 0x1 ref 1 00000100:00100000:3.0:1789760731.324718:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720592368164 time 720655647836 state 0x0 ref 1 00000100:00100000:3.0:1789760731.324735:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484203776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760731.324801:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20488192, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:3.0:1789760731.324813:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.324835:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:3.0:1789760731.324841:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760731.324851:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20488192->20492287] (req 20488192->20492287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.324867:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20488192->20492287] (req 20488192->20492287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760731.381863:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972320 is committed 00002000:00010000:3.0:1789760731.382273:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20488192 -> 20492288 00010000:00010000:3.0:1789760731.382294:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20488192->20492287] (req 20488192->20492287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.382310:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20488192->20492287] (req 20488192->20492287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.382320:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20488192->20492287] (req 20488192->20492287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.382328:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760731.382335:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20488192->20492287] (req 20488192->20492287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.382345:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20488192->20492287] (req 20488192->20492287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.382371:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20488192->20492287] (req 20488192->20492287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.382381:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208283 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20488192->20492287] (req 20488192->20492287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760731.382695:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484203776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57956us (58300us total) trans 4294972320 rc 0/0 00000100:00100000:3.0:1789760731.382709:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5031 00000100:00100000:3.0:1789760731.382722:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720592368164 time 720655647836 state 0x2 ref 0 00000100:00100000:1.0:1789760731.389449:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760731.395349:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484203904 00000100:00100000:2.0:1789760731.395374:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484203904 00000100:00100000:2.0:1789760731.395543:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720592368164 time 720655647836 state 0x0 ref 1 00000100:00100000:2.0:1789760731.395568:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720655908252 time 720655647836 state 0x1 ref 1 00000100:00100000:2.0:1789760731.395613:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720655908252 time 720726543697 state 0x0 ref 1 00000100:00100000:2.0:1789760731.395632:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484203904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760731.395702:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20492288, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:2.0:1789760731.395713:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.395733:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:2.0:1789760731.395739:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760731.395747:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20492288->20496383] (req 20492288->20496383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.395765:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20492288->20496383] (req 20492288->20496383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760731.467813:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972321 is committed 00002000:00010000:2.0:1789760731.467952:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20492288 -> 20496384 00010000:00010000:2.0:1789760731.467972:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20492288->20496383] (req 20492288->20496383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.467987:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20492288->20496383] (req 20492288->20496383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.468002:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20492288->20496383] (req 20492288->20496383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.468010:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.468017:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20492288->20496383] (req 20492288->20496383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.468027:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20492288->20496383] (req 20492288->20496383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.468051:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20492288->20496383] (req 20492288->20496383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.468059:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620828a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20492288->20496383] (req 20492288->20496383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.468375:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484203904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 72743us (78934us total) trans 4294972321 rc 0/0 00000100:00100000:2.0:1789760731.468399:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5032 00000100:00100000:2.0:1789760731.468413:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720655908252 time 720726543697 state 0x2 ref 0 00000100:00100000:0.0:1789760731.473950:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760731.474048:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484204032 00000100:00100000:2.0:1789760731.474061:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484204032 00000100:00100000:2.0:1789760731.474424:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720655908252 time 720726543697 state 0x0 ref 1 00000100:00100000:2.0:1789760731.474445:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720726804113 time 720726543697 state 0x1 ref 1 00000100:00100000:3.0:1789760731.474620:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720726804113 time 720805547829 state 0x0 ref 1 00000100:00100000:3.0:1789760731.474638:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484204032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760731.474703:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20496384, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:3.0:1789760731.474716:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.474737:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:3.0:1789760731.474743:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760731.474750:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20496384->20500479] (req 20496384->20500479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.474767:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20496384->20500479] (req 20496384->20500479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760731.500665:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972322 is committed 00002000:00010000:0.0:1789760731.500761:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20496384 -> 20500480 00010000:00010000:0.0:1789760731.500781:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20496384->20500479] (req 20496384->20500479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.500798:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20496384->20500479] (req 20496384->20500479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.500809:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20496384->20500479] (req 20496384->20500479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.500817:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760731.500824:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20496384->20500479] (req 20496384->20500479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.500835:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20496384->20500479] (req 20496384->20500479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.500861:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20496384->20500479] (req 20496384->20500479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.500869:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208291 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20496384->20500479] (req 20496384->20500479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760731.501004:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484204032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26363us (27062us total) trans 4294972322 rc 0/0 00000100:00100000:0.0:1789760731.501021:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5033 00000100:00100000:0.0:1789760731.501035:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720726804113 time 720805547829 state 0x2 ref 0 00000100:00100000:0.0:1789760731.504302:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760731.504551:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484204160 00000100:00100000:3.0:1789760731.504565:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484204160 00000100:00100000:3.0:1789760731.504706:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720726804113 time 720805547829 state 0x0 ref 1 00000100:00100000:3.0:1789760731.504725:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720805808245 time 720805547829 state 0x1 ref 1 00000100:00100000:3.0:1789760731.504763:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720805808245 time 720835693557 state 0x0 ref 1 00000100:00100000:3.0:1789760731.504779:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484204160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760731.504847:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20500480, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:3.0:1789760731.504859:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.504878:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:3.0:1789760731.504884:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760731.504891:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20500480->20504575] (req 20500480->20504575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.505570:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20500480->20504575] (req 20500480->20504575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.533234:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df82e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508915584:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760731.533336:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760731.533448:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508915584 00000100:00100000:2.0:1789760731.533458:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508915584 00000100:00100000:2.0:1789760731.533540:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 372 00000100:00100000:2.0:1789760731.533552:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508915584:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1789760731.533597:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760731.533651:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508915584:12345-0@lo:13:osp-pre-1-0.0 Request processed in 101us (315us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760731.533658:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 372 00000100:00080000:2.0:1789760731.533681:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1789760731.533725:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df82e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508915584:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760731.534013:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1789760731.534025:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508915712:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760731.534061:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760731.534120:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508915712 00000100:00100000:0.0:1789760731.534126:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508915712 00000100:00100000:0.0:1789760731.534161:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 373 00000100:00100000:0.0:1789760731.534179:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508915712:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:0.0:1789760731.534193:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:0.0:1789760731.534225:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508915712:12345-0@lo:13:osp-pre-1-1.0 Request processed in 57us (163us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760731.534233:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 373 00000100:00100000:0.0:1789760731.534265:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508915712:0@lo:13:osp-pre-1-1.0 00000100:00080000:2.0:1789760731.534378:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:1.0:1789760731.535454:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c886ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508915968:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.535531:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760731.535563:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501b82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508916480:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.535586:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760731.535614:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501b81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508916992:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.535636:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760731.535659:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1789760731.535665:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c886e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508916096:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.535690:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760731.535708:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501b81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508916608:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.535730:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760731.535746:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501b80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508917120:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.535764:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760731.535792:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508915968 00000100:00100000:1.0:1789760731.535800:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508915968 00000100:00100000:1.0:1789760731.535843:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508916480 00000100:00100000:1.0:1789760731.535846:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508916480 00000100:00100000:1.0:1789760731.536019:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508916992 00000100:00100000:1.0:1789760731.536023:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508916992 00000100:00100000:1.0:1789760731.536040:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508916608 00000100:00100000:1.0:1789760731.536042:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508916608 00000100:00100000:1.0:1789760731.536055:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508917120 00000100:00100000:1.0:1789760731.536058:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508917120 00000100:00100000:1.0:1789760731.536110:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 895 00000100:00100000:1.0:1789760731.536120:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508915968:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.536250:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508915968:12345-0@lo:400:kworker.0 Request processed in 129us (718us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760731.536258:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 895 00000100:00100000:1.0:1789760731.536277:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 896 00000100:00100000:1.0:1789760731.536283:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508916480:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.536311:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508916480:12345-0@lo:400:kworker.0 Request processed in 28us (724us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760731.536316:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 896 00000100:00100000:1.0:1789760731.536325:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 897 00000100:00100000:1.0:1789760731.536330:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b82d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508916992:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.536354:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b82d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508916992:12345-0@lo:400:kworker.0 Request processed in 24us (717us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760731.536360:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 897 00000100:00100000:1.0:1789760731.536372:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 898 00000100:00100000:1.0:1789760731.536378:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508916608:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.536414:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508916608:12345-0@lo:400:kworker.0 Request processed in 37us (683us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760731.536420:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 898 00000100:00100000:1.0:1789760731.536432:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 899 00000100:00100000:1.0:1789760731.536437:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b82a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508917120:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.536464:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b82a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508917120:12345-0@lo:400:kworker.0 Request processed in 27us (698us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760731.536470:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 899 00000100:00100000:1.0:1789760731.536498:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508916096 00000100:00100000:1.0:1789760731.536502:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508916096 00000100:00100000:3.0:1789760731.536618:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c886f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0001_UUID:3480:1876699508916224:0@lo:400:kworker.0 00000100:00100000:3.0:1789760731.536654:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760731.536708:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 431 00000100:00100000:1.0:1789760731.536715:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508916096:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.536748:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508916096:12345-0@lo:400:kworker.0 Request processed in 34us (1057us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760731.536754:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 431 00000100:00100000:1.0:1789760731.536770:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508916224 00000100:00100000:1.0:1789760731.536773:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508916224 00000100:00100000:1.0:1789760731.536792:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 900 00000100:00100000:1.0:1789760731.536797:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3480:x1876699508916224:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.536821:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3480:x1876699508916224:12345-0@lo:400:kworker.0 Request processed in 24us (167us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760731.536827:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 900 00000100:00080000:1.0:1789760731.536926:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:1.0:1789760731.536931:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:1.0:1789760731.536971:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c886e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508916096:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.536984:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501b81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508916608:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.536993:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501b80000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508917120:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.537004:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c886ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508915968:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.537014:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501b82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508916480:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.537023:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501b81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508916992:0@lo:400:kworker.0 00000100:00100000:0.0:1789760731.537374:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508915840:0@lo:400:kworker.0 00000100:00100000:0.0:1789760731.537426:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760731.537484:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508915840 00000100:00100000:1.0:1789760731.537488:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508915840 00000100:00100000:3.0:1789760731.538787:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c886f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0001_UUID:3480:1876699508916224:0@lo:400:kworker.0 00000100:00100000:3.0:1789760731.538803:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501b80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508916736:0@lo:400:kworker.0 00000100:00100000:3.0:1789760731.538988:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760731.539156:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c886d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508916352:0@lo:400:kworker.0 00000100:00100000:0.0:1789760731.539204:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760731.539225:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501b80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508916864:0@lo:400:kworker.0 00000100:00100000:0.0:1789760731.539245:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760731.539285:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508916736 00000100:00100000:0.0:1789760731.539292:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508916736 00000100:00100000:0.0:1789760731.539353:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508916864 00000100:00100000:0.0:1789760731.539357:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508916864 00000100:00100000:0.0:1789760731.539402:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 314 00000100:00100000:0.0:1789760731.539414:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508916736:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760731.539474:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508916736:12345-0@lo:400:kworker.0 Request processed in 63us (487us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760731.539482:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 314 00000100:00100000:0.0:1789760731.539498:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 315 00000100:00100000:0.0:1789760731.539505:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508916864:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760731.539656:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508916864:12345-0@lo:400:kworker.0 Request processed in 152us (409us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760731.539663:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 315 00000100:00100000:0.0:1789760731.539701:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501b80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508916736:0@lo:400:kworker.0 00000100:00100000:0.0:1789760731.539727:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501b80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508916864:0@lo:400:kworker.0 00000100:00100000:3.0:1789760731.539990:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508916352 00000100:00100000:3.0:1789760731.539996:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508916352 00000100:00100000:3.0:1789760731.540041:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 432 00000100:00100000:3.0:1789760731.540048:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508916352:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760731.540117:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508916352:12345-0@lo:400:kworker.0 Request processed in 70us (912us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760731.540125:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 432 00000100:00100000:0.0:1789760731.540380:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c886d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508916352:0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.541156:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 474 00000100:00100000:1.0:1789760731.541169:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b7450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508915840:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760731.541410:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b7450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508915840:12345-0@lo:400:kworker.0 Request processed in 244us (3982us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760731.541418:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 474 00000100:00080000:1.0:1789760731.541472:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760731.541726:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508915840:0@lo:400:kworker.0 00000001:00080000:1.0:1789760731.544305:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972323 is committed 00002000:00010000:2.0:1789760731.544644:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20500480 -> 20504576 00010000:00010000:2.0:1789760731.544665:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20500480->20504575] (req 20500480->20504575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.544680:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20500480->20504575] (req 20500480->20504575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.544690:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20500480->20504575] (req 20500480->20504575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.544698:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.544704:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20500480->20504575] (req 20500480->20504575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.544716:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20500480->20504575] (req 20500480->20504575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.544742:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20500480->20504575] (req 20500480->20504575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.544751:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208298 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20500480->20504575] (req 20500480->20504575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.544874:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484204160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40092us (40579us total) trans 4294972323 rc 0/0 00000100:00100000:2.0:1789760731.544888:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5034 00000100:00100000:2.0:1789760731.544900:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720805808245 time 720835693557 state 0x2 ref 0 00000100:00100000:1.0:1789760731.548213:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760731.548312:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484204288 00000100:00100000:2.0:1789760731.548325:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484204288 00000100:00100000:2.0:1789760731.548412:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720805808245 time 720835693557 state 0x0 ref 1 00000100:00100000:2.0:1789760731.548430:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720835953973 time 720835693557 state 0x1 ref 1 00000100:00100000:2.0:1789760731.548463:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720835953973 time 720879393510 state 0x0 ref 1 00000100:00100000:2.0:1789760731.548477:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484204288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760731.548541:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20504576, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:2.0:1789760731.548552:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.548573:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:2.0:1789760731.548579:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760731.548587:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20504576->20508671] (req 20504576->20508671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.548602:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20504576->20508671] (req 20504576->20508671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760731.598469:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972324 is committed 00002000:00010000:2.0:1789760731.598760:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20504576 -> 20508672 00010000:00010000:2.0:1789760731.598968:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20504576->20508671] (req 20504576->20508671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.598987:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20504576->20508671] (req 20504576->20508671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.598998:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20504576->20508671] (req 20504576->20508671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.599007:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.599013:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20504576->20508671] (req 20504576->20508671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.599023:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20504576->20508671] (req 20504576->20508671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.599050:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20504576->20508671] (req 20504576->20508671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.599059:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620829f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20504576->20508671] (req 20504576->20508671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.599484:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484204288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51001us (51278us total) trans 4294972324 rc 0/0 00000100:00100000:2.0:1789760731.599497:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5035 00000100:00100000:2.0:1789760731.599510:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720835953973 time 720879393510 state 0x2 ref 0 00000100:00100000:0.0:1789760731.606997:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760731.607244:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484204416 00000100:00100000:2.0:1789760731.607259:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484204416 00000100:00100000:2.0:1789760731.607332:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720835953973 time 720879393510 state 0x0 ref 1 00000100:00100000:2.0:1789760731.607350:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720879653926 time 720879393510 state 0x1 ref 1 00000100:00100000:2.0:1789760731.607386:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720879653926 time 720938316907 state 0x0 ref 1 00000100:00100000:2.0:1789760731.607411:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484204416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760731.607474:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20508672, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:2.0:1789760731.607485:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.607505:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=720s v=(0 0) 00010000:00010000:2.0:1789760731.607511:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760731.607528:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20508672->20512767] (req 20508672->20512767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.607550:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20508672->20512767] (req 20508672->20512767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760731.669054:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972325 is committed 00002000:00010000:2.0:1789760731.669187:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20508672 -> 20512768 00010000:00010000:2.0:1789760731.669207:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20508672->20512767] (req 20508672->20512767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.669223:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20508672->20512767] (req 20508672->20512767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.669234:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20508672->20512767] (req 20508672->20512767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.669243:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.669249:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20508672->20512767] (req 20508672->20512767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.669261:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20508672->20512767] (req 20508672->20512767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.669288:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20508672->20512767] (req 20508672->20512767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.669297:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082a6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20508672->20512767] (req 20508672->20512767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.669434:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484204416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62019us (62446us total) trans 4294972325 rc 0/0 00000100:00100000:2.0:1789760731.669447:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5036 00000100:00100000:2.0:1789760731.669460:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720879653926 time 720938316907 state 0x2 ref 0 00000100:00100000:0.0:1789760731.674410:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760731.674662:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484204544 00000100:00100000:3.0:1789760731.674682:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484204544 00000100:00100000:3.0:1789760731.674835:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720879653926 time 720938316907 state 0x0 ref 1 00000100:00100000:3.0:1789760731.674855:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720938577323 time 720938316907 state 0x1 ref 1 00000100:00100000:3.0:1789760731.674894:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720938577323 time 721005823922 state 0x0 ref 1 00000100:00100000:3.0:1789760731.674910:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484204544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760731.674987:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20512768, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:3.0:1789760731.674998:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.675020:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:3.0:1789760731.675025:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760731.675033:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20512768->20516863] (req 20512768->20516863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.675061:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20512768->20516863] (req 20512768->20516863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760731.722217:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972326 is committed 00002000:00010000:0.0:1789760731.722325:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20512768 -> 20516864 00010000:00010000:0.0:1789760731.722346:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20512768->20516863] (req 20512768->20516863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.722362:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20512768->20516863] (req 20512768->20516863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.722372:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20512768->20516863] (req 20512768->20516863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.722380:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760731.722385:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20512768->20516863] (req 20512768->20516863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.722404:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20512768->20516863] (req 20512768->20516863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.722429:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20512768->20516863] (req 20512768->20516863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.722437:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962082ad lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20512768->20516863] (req 20512768->20516863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760731.722742:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484204544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47829us (48351us total) trans 4294972326 rc 0/0 00000100:00100000:0.0:1789760731.722755:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5037 00000100:00100000:0.0:1789760731.722767:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720938577323 time 721005823922 state 0x2 ref 0 00000100:00100000:3.0:1789760731.729499:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760731.729624:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484204672 00000100:00100000:2.0:1789760731.729639:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484204672 00000100:00100000:2.0:1789760731.729716:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 720938577323 time 721005823922 state 0x0 ref 1 00000100:00100000:2.0:1789760731.729736:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721006084338 time 721005823922 state 0x1 ref 1 00000100:00100000:2.0:1789760731.729773:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721006084338 time 721060702868 state 0x0 ref 1 00000100:00100000:2.0:1789760731.729788:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484204672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760731.729865:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20516864, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:2.0:1789760731.729875:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.729896:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:2.0:1789760731.729912:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760731.729920:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20516864->20520959] (req 20516864->20520959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.729945:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20516864->20520959] (req 20516864->20520959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760731.766971:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972327 is committed 00002000:00010000:2.0:1789760731.771222:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20516864 -> 20520960 00010000:00010000:2.0:1789760731.771250:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20516864->20520959] (req 20516864->20520959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.771265:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20516864->20520959] (req 20516864->20520959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.771275:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20516864->20520959] (req 20516864->20520959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.771282:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.771288:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20516864->20520959] (req 20516864->20520959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.771298:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20516864->20520959] (req 20516864->20520959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.771324:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20516864->20520959] (req 20516864->20520959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.771332:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082b4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20516864->20520959] (req 20516864->20520959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.771520:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484204672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41729us (42030us total) trans 4294972327 rc 0/0 00000100:00100000:2.0:1789760731.771537:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5038 00000100:00100000:2.0:1789760731.771549:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721006084338 time 721060702868 state 0x2 ref 0 00000100:00100000:1.0:1789760731.778617:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760731.778770:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484204800 00000100:00100000:2.0:1789760731.778785:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484204800 00000100:00100000:2.0:1789760731.778847:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721006084338 time 721060702868 state 0x0 ref 1 00000100:00100000:2.0:1789760731.778862:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721060963284 time 721060702868 state 0x1 ref 1 00000100:00100000:2.0:1789760731.778972:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721060963284 time 721109892790 state 0x0 ref 1 00000100:00100000:2.0:1789760731.778986:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484204800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760731.779033:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20520960, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:2.0:1789760731.779042:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.779057:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:2.0:1789760731.779061:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760731.779102:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20520960->20525055] (req 20520960->20525055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.779116:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20520960->20525055] (req 20520960->20525055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760731.805377:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972328 is committed 00002000:00010000:2.0:1789760731.805543:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20520960 -> 20525056 00010000:00010000:2.0:1789760731.805558:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20520960->20525055] (req 20520960->20525055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.805573:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20520960->20525055] (req 20520960->20525055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.805582:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20520960->20525055] (req 20520960->20525055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.805589:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.805594:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20520960->20525055] (req 20520960->20525055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.805604:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20520960->20525055] (req 20520960->20525055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.805624:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20520960->20525055] (req 20520960->20525055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.806271:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082bb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20520960->20525055] (req 20520960->20525055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.806422:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484204800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27435us (27815us total) trans 4294972328 rc 0/0 00000100:00100000:2.0:1789760731.806431:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5039 00000100:00100000:2.0:1789760731.806441:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721060963284 time 721109892790 state 0x2 ref 0 00000100:00100000:1.0:1789760731.814244:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760731.814409:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484204928 00000100:00100000:2.0:1789760731.814419:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484204928 00000100:00100000:2.0:1789760731.814470:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721060963284 time 721109892790 state 0x0 ref 1 00000100:00100000:2.0:1789760731.814486:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721110153206 time 721109892790 state 0x1 ref 1 00000100:00100000:2.0:1789760731.814555:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721110153206 time 721145485700 state 0x0 ref 1 00000100:00100000:2.0:1789760731.814568:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484204928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760731.814628:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20525056, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:2.0:1789760731.814637:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.814652:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:2.0:1789760731.814656:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760731.814662:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20525056->20529151] (req 20525056->20529151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.814675:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20525056->20529151] (req 20525056->20529151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760731.845731:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20525056 -> 20529152 00010000:00010000:2.0:1789760731.845749:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20525056->20529151] (req 20525056->20529151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.845762:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20525056->20529151] (req 20525056->20529151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.845778:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20525056->20529151] (req 20525056->20529151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.845785:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.845790:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20525056->20529151] (req 20525056->20529151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.845799:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20525056->20529151] (req 20525056->20529151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.845819:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20525056->20529151] (req 20525056->20529151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.845828:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082c2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20525056->20529151] (req 20525056->20529151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.845919:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484204928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31349us (31681us total) trans 4294972329 rc 0/0 00000100:00100000:2.0:1789760731.845929:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5040 00000100:00100000:2.0:1789760731.845939:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721110153206 time 721145485700 state 0x2 ref 0 00000001:00080000:1.0:1789760731.846478:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972329 is committed 00000100:00100000:3.0:1789760731.850456:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760731.850591:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484205056 00000100:00100000:0.0:1789760731.850600:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484205056 00000100:00100000:0.0:1789760731.850658:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721110153206 time 721145485700 state 0x0 ref 1 00000100:00100000:0.0:1789760731.850673:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721145746116 time 721145485700 state 0x1 ref 1 00000100:00100000:0.0:1789760731.850739:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721145746116 time 721181669063 state 0x0 ref 1 00000100:00100000:0.0:1789760731.850753:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706fc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484205056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760731.850802:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20529152, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:0.0:1789760731.850811:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.850828:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:0.0:1789760731.850832:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760731.850839:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20529152->20533247] (req 20529152->20533247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.850853:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20529152->20533247] (req 20529152->20533247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760731.878315:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20529152 -> 20533248 00010000:00010000:1.0:1789760731.878335:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20529152->20533247] (req 20529152->20533247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.878347:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20529152->20533247] (req 20529152->20533247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.878362:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20529152->20533247] (req 20529152->20533247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.878370:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760731.878374:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20529152->20533247] (req 20529152->20533247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.878384:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20529152->20533247] (req 20529152->20533247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.878413:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20529152->20533247] (req 20529152->20533247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.878420:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082c9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20529152->20533247] (req 20529152->20533247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760731.878526:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706fc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484205056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27769us (28074us total) trans 4294972330 rc 0/0 00000100:00100000:1.0:1789760731.878537:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5041 00000100:00100000:1.0:1789760731.878546:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721145746116 time 721181669063 state 0x2 ref 0 00000001:00080000:1.0:1789760731.878832:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972330 is committed 00000100:00100000:0.0:1789760731.883022:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760731.883385:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484205184 00000100:00100000:3.0:1789760731.883404:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484205184 00000100:00100000:3.0:1789760731.883457:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721145746116 time 721181669063 state 0x0 ref 1 00000100:00100000:3.0:1789760731.883471:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721181929479 time 721181669063 state 0x1 ref 1 00000100:00100000:3.0:1789760731.883509:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721181929479 time 721214440649 state 0x0 ref 1 00000100:00100000:3.0:1789760731.883522:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484205184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760731.883571:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20533248, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:3.0:1789760731.883579:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.883596:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:3.0:1789760731.883600:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760731.883606:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20533248->20537343] (req 20533248->20537343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.883619:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20533248->20537343] (req 20533248->20537343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760731.911470:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972331 is committed 00002000:00010000:3.0:1789760731.911695:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20533248 -> 20537344 00010000:00010000:3.0:1789760731.911712:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20533248->20537343] (req 20533248->20537343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.911724:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20533248->20537343] (req 20533248->20537343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.911734:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20533248->20537343] (req 20533248->20537343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.911740:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760731.911744:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20533248->20537343] (req 20533248->20537343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.911753:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20533248->20537343] (req 20533248->20537343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.911772:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20533248->20537343] (req 20533248->20537343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760731.911780:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082d0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20533248->20537343] (req 20533248->20537343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760731.911870:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484205184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28346us (28853us total) trans 4294972331 rc 0/0 00000100:00100000:3.0:1789760731.911884:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5042 00000100:00100000:3.0:1789760731.911893:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721181929479 time 721214440649 state 0x2 ref 0 00000100:00100000:0.0:1789760731.915041:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760731.915201:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484205312 00000100:00100000:1.0:1789760731.915211:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484205312 00000100:00100000:1.0:1789760731.915264:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721181929479 time 721214440649 state 0x0 ref 1 00000100:00100000:1.0:1789760731.915282:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721214701065 time 721214440649 state 0x1 ref 1 00000100:00100000:1.0:1789760731.915308:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721214701065 time 721246239135 state 0x0 ref 1 00000100:00100000:1.0:1789760731.915321:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484205312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760731.915371:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20537344, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:1.0:1789760731.915380:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.915407:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:1.0:1789760731.915412:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760731.915418:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20537344->20541439] (req 20537344->20541439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760731.915431:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20537344->20541439] (req 20537344->20541439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760731.946935:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972332 is committed 00002000:00010000:0.0:1789760731.947039:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20537344 -> 20541440 00010000:00010000:0.0:1789760731.947057:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20537344->20541439] (req 20537344->20541439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.947099:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20537344->20541439] (req 20537344->20541439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.947108:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20537344->20541439] (req 20537344->20541439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.947115:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760731.947120:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20537344->20541439] (req 20537344->20541439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.947129:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20537344->20541439] (req 20537344->20541439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.947152:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20537344->20541439] (req 20537344->20541439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.947160:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962082d7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20537344->20541439] (req 20537344->20541439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760731.947278:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484205312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31953us (32237us total) trans 4294972332 rc 0/0 00000100:00100000:0.0:1789760731.947290:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5043 00000100:00100000:0.0:1789760731.947302:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721214701065 time 721246239135 state 0x2 ref 0 00000100:00100000:1.0:1789760731.951622:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760731.951724:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484205440 00000100:00100000:2.0:1789760731.951735:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484205440 00000100:00100000:2.0:1789760731.951796:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721214701065 time 721246239135 state 0x0 ref 1 00000100:00100000:2.0:1789760731.951814:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721246499551 time 721246239135 state 0x1 ref 1 00000100:00100000:2.0:1789760731.951839:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721246499551 time 721282770446 state 0x0 ref 1 00000100:00100000:2.0:1789760731.951852:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484205440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760731.951905:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20541440, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:2.0:1789760731.951914:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.951931:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:2.0:1789760731.951936:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760731.951942:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20541440->20545535] (req 20541440->20545535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.951957:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20541440->20545535] (req 20541440->20545535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760731.978162:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972333 is committed 00002000:00010000:2.0:1789760731.978329:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20541440 -> 20545536 00010000:00010000:2.0:1789760731.978343:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20541440->20545535] (req 20541440->20545535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.978356:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20541440->20545535] (req 20541440->20545535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.978365:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20541440->20545535] (req 20541440->20545535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.978372:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760731.978378:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20541440->20545535] (req 20541440->20545535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.978394:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20541440->20545535] (req 20541440->20545535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.978413:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20541440->20545535] (req 20541440->20545535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760731.978421:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962082de lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20541440->20545535] (req 20541440->20545535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760731.978576:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484205440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26721us (26960us total) trans 4294972333 rc 0/0 00000100:00100000:2.0:1789760731.978587:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5044 00000100:00100000:2.0:1789760731.978596:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721246499551 time 721282770446 state 0x2 ref 0 00000100:00100000:1.0:1789760731.982562:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760731.982795:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484205568 00000100:00100000:0.0:1789760731.982804:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484205568 00000100:00100000:0.0:1789760731.982864:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721246499551 time 721282770446 state 0x0 ref 1 00000100:00100000:0.0:1789760731.982879:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721283030862 time 721282770446 state 0x1 ref 1 00000100:00100000:0.0:1789760731.982945:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721283030862 time 721313875088 state 0x0 ref 1 00000100:00100000:0.0:1789760731.982959:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484205568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760731.983008:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20545536, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:0.0:1789760731.983017:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.983032:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:0.0:1789760731.983036:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760731.983042:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20545536->20549631] (req 20545536->20549631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760731.983054:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20545536->20549631] (req 20545536->20549631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760732.014916:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972334 is committed 00002000:00010000:0.0:1789760732.015015:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20545536 -> 20549632 00010000:00010000:0.0:1789760732.015029:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20545536->20549631] (req 20545536->20549631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.015043:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20545536->20549631] (req 20545536->20549631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.015051:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20545536->20549631] (req 20545536->20549631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.015058:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760732.015062:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20545536->20549631] (req 20545536->20549631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.015106:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20545536->20549631] (req 20545536->20549631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.015125:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20545536->20549631] (req 20545536->20549631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.015132:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082e5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20545536->20549631] (req 20545536->20549631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760732.015225:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484205568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32266us (32668us total) trans 4294972334 rc 0/0 00000100:00100000:0.0:1789760732.015239:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5045 00000100:00100000:0.0:1789760732.015248:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721283030862 time 721313875088 state 0x2 ref 0 00000100:00100000:0.0:1789760732.018462:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760732.018586:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484205696 00000100:00100000:3.0:1789760732.018596:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484205696 00000100:00100000:3.0:1789760732.018664:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721283030862 time 721313875088 state 0x0 ref 1 00000100:00100000:3.0:1789760732.018690:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721314135504 time 721313875088 state 0x1 ref 1 00000100:00100000:3.0:1789760732.018720:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721314135504 time 721349651504 state 0x0 ref 1 00000100:00100000:3.0:1789760732.018733:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484205696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760732.018777:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20549632, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:3.0:1789760732.018786:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.018801:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:3.0:1789760732.018805:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760732.018811:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20549632->20553727] (req 20549632->20553727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.018824:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20549632->20553727] (req 20549632->20553727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1789760732.019558:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00010000:00010000:0.0:1789760732.044450:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760732.047621:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972335 is committed 00002000:00010000:1.0:1789760732.047696:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20549632 -> 20553728 00010000:00010000:1.0:1789760732.047711:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20549632->20553727] (req 20549632->20553727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.047726:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20549632->20553727] (req 20549632->20553727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.047735:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20549632->20553727] (req 20549632->20553727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.047743:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760732.047748:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20549632->20553727] (req 20549632->20553727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.047756:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20549632->20553727] (req 20549632->20553727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.047776:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20549632->20553727] (req 20549632->20553727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.047784:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962082ec lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20549632->20553727] (req 20549632->20553727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760732.047873:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484205696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29136us (29414us total) trans 4294972335 rc 0/0 00000100:00100000:1.0:1789760732.047883:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5046 00000100:00100000:1.0:1789760732.047892:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721314135504 time 721349651504 state 0x2 ref 0 00000100:00100000:1.0:1789760732.051400:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760732.051672:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484205824 00000100:00100000:0.0:1789760732.051747:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484205824 00000100:00100000:0.0:1789760732.051807:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721314135504 time 721349651504 state 0x0 ref 1 00000100:00100000:0.0:1789760732.051823:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721349911920 time 721349651504 state 0x1 ref 1 00000100:00100000:0.0:1789760732.051855:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721349911920 time 721382786622 state 0x0 ref 1 00000100:00100000:0.0:1789760732.051868:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484205824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760732.051913:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20553728, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:0.0:1789760732.051921:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.051936:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:0.0:1789760732.051940:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760732.051946:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20553728->20557823] (req 20553728->20557823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.051958:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20553728->20557823] (req 20553728->20557823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760732.083736:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20553728 -> 20557824 00010000:00010000:1.0:1789760732.083756:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20553728->20557823] (req 20553728->20557823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.083769:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20553728->20557823] (req 20553728->20557823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.083778:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20553728->20557823] (req 20553728->20557823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.083786:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760732.083791:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20553728->20557823] (req 20553728->20557823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.083800:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20553728->20557823] (req 20553728->20557823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.083819:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20553728->20557823] (req 20553728->20557823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.083828:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962082f3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20553728->20557823] (req 20553728->20557823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760732.083951:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484205824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32079us (32565us total) trans 4294972336 rc 0/0 00000100:00100000:1.0:1789760732.083961:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5047 00000100:00100000:1.0:1789760732.083971:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721349911920 time 721382786622 state 0x2 ref 0 00000001:00080000:1.0:1789760732.084009:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972336 is committed 00000100:00100000:1.0:1789760732.087966:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760732.088055:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484205952 00000100:00100000:0.0:1789760732.088270:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484205952 00000100:00100000:0.0:1789760732.088332:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721349911920 time 721382786622 state 0x0 ref 1 00000100:00100000:0.0:1789760732.088347:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721383047038 time 721382786622 state 0x1 ref 1 00000100:00100000:0.0:1789760732.088375:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721383047038 time 721419306497 state 0x0 ref 1 00000100:00100000:0.0:1789760732.088399:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484205952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760732.088447:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20557824, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:0.0:1789760732.088456:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.088471:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:0.0:1789760732.088568:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760732.088575:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20557824->20561919] (req 20557824->20561919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.088590:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20557824->20561919] (req 20557824->20561919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760732.114429:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972337 is committed 00002000:00010000:0.0:1789760732.114540:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20557824 -> 20561920 00010000:00010000:0.0:1789760732.114558:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20557824->20561919] (req 20557824->20561919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.114576:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20557824->20561919] (req 20557824->20561919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.114587:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20557824->20561919] (req 20557824->20561919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.114594:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760732.114599:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20557824->20561919] (req 20557824->20561919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.114607:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20557824->20561919] (req 20557824->20561919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.114629:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20557824->20561919] (req 20557824->20561919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.114636:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962082fa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20557824->20561919] (req 20557824->20561919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760732.114738:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484205952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26348us (26776us total) trans 4294972337 rc 0/0 00000100:00100000:0.0:1789760732.114748:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5048 00000100:00100000:0.0:1789760732.114757:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721383047038 time 721419306497 state 0x2 ref 0 00000100:00100000:1.0:1789760732.118640:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760732.118980:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484206080 00000100:00100000:0.0:1789760732.118991:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484206080 00000100:00100000:0.0:1789760732.119164:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721383047038 time 721419306497 state 0x0 ref 1 00000100:00100000:0.0:1789760732.119249:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721419566913 time 721419306497 state 0x1 ref 1 00000100:00100000:0.0:1789760732.119314:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721419566913 time 721450244280 state 0x0 ref 1 00000100:00100000:0.0:1789760732.119328:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484206080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760732.119376:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20561920, mtime: 0x6aad94db, atime: 0x0, ctime: 0x6aad94db, blocks: 0xa007 00010000:00010000:0.0:1789760732.119394:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.119409:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:0.0:1789760732.119413:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760732.119419:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20561920->20566015] (req 20561920->20566015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.119432:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20561920->20566015] (req 20561920->20566015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760732.150958:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972338 is committed 00002000:00010000:1.0:1789760732.151043:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20561920 -> 20566016 00002000:00010000:1.0:1789760732.151048:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760731 -> 1789760732 00002000:00010000:1.0:1789760732.151052:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760731 -> 1789760732 00010000:00010000:1.0:1789760732.151087:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20561920->20566015] (req 20561920->20566015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.151100:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20561920->20566015] (req 20561920->20566015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.151110:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20561920->20566015] (req 20561920->20566015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.151117:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760732.151122:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20561920->20566015] (req 20561920->20566015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.151132:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20561920->20566015] (req 20561920->20566015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.151153:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20561920->20566015] (req 20561920->20566015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.151162:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208301 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20561920->20566015] (req 20561920->20566015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760732.151268:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484206080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31939us (32633us total) trans 4294972338 rc 0/0 00000100:00100000:1.0:1789760732.151278:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5049 00000100:00100000:1.0:1789760732.151287:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721419566913 time 721450244280 state 0x2 ref 0 00000100:00100000:1.0:1789760732.156728:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760732.156805:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484206208 00000100:00100000:3.0:1789760732.156825:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484206208 00000100:00100000:3.0:1789760732.156880:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721419566913 time 721450244280 state 0x0 ref 1 00000100:00100000:3.0:1789760732.156895:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721450504696 time 721450244280 state 0x1 ref 1 00000100:00100000:3.0:1789760732.156927:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721450504696 time 721487858035 state 0x0 ref 1 00000100:00100000:3.0:1789760732.156940:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484206208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760732.156985:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20566016, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:3.0:1789760732.156994:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.157010:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:3.0:1789760732.157015:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760732.157021:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20566016->20570111] (req 20566016->20570111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.157036:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20566016->20570111] (req 20566016->20570111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760732.184936:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972339 is committed 00002000:00010000:3.0:1789760732.185164:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20566016 -> 20570112 00010000:00010000:3.0:1789760732.185238:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20566016->20570111] (req 20566016->20570111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.185252:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20566016->20570111] (req 20566016->20570111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.185267:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20566016->20570111] (req 20566016->20570111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.185275:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760732.185280:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20566016->20570111] (req 20566016->20570111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.185290:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20566016->20570111] (req 20566016->20570111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.185310:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20566016->20570111] (req 20566016->20570111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.185318:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208308 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20566016->20570111] (req 20566016->20570111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760732.185422:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484206208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28479us (28699us total) trans 4294972339 rc 0/0 00000100:00100000:3.0:1789760732.185432:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5050 00000100:00100000:3.0:1789760732.185442:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721450504696 time 721487858035 state 0x2 ref 0 00000100:00100000:3.0:1789760732.189913:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760732.190009:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484206336 00000100:00100000:1.0:1789760732.190019:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484206336 00000100:00100000:1.0:1789760732.190618:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721450504696 time 721487858035 state 0x0 ref 1 00000100:00100000:1.0:1789760732.190635:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721488118451 time 721487858035 state 0x1 ref 1 00000100:00100000:1.0:1789760732.190670:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721488118451 time 721521601067 state 0x0 ref 1 00000100:00100000:1.0:1789760732.190685:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484206336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760732.190736:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20570112, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:1.0:1789760732.190744:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.190763:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:1.0:1789760732.190768:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760732.190776:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20570112->20574207] (req 20570112->20574207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.190798:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20570112->20574207] (req 20570112->20574207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760732.248179:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972340 is committed 00002000:00010000:3.0:1789760732.248323:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20570112 -> 20574208 00010000:00010000:3.0:1789760732.248338:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20570112->20574207] (req 20570112->20574207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.248356:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20570112->20574207] (req 20570112->20574207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.248365:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20570112->20574207] (req 20570112->20574207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.248373:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760732.248377:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20570112->20574207] (req 20570112->20574207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.248398:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20570112->20574207] (req 20570112->20574207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.248419:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20570112->20574207] (req 20570112->20574207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.248427:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620830f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20570112->20574207] (req 20570112->20574207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760732.248610:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484206336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57925us (58702us total) trans 4294972340 rc 0/0 00000100:00100000:3.0:1789760732.248619:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5051 00000100:00100000:3.0:1789760732.248629:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721488118451 time 721521601067 state 0x2 ref 0 00000100:00100000:1.0:1789760732.254547:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760732.254638:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484206464 00000100:00100000:0.0:1789760732.254650:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484206464 00000100:00100000:0.0:1789760732.254709:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721488118451 time 721521601067 state 0x0 ref 1 00000100:00100000:0.0:1789760732.254724:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721521861483 time 721521601067 state 0x1 ref 1 00000100:00100000:0.0:1789760732.254755:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721521861483 time 721585685489 state 0x0 ref 1 00000100:00100000:0.0:1789760732.254768:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484206464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760732.254821:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20574208, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:0.0:1789760732.254830:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.254845:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:0.0:1789760732.254849:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760732.254856:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20574208->20578303] (req 20574208->20578303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.254870:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20574208->20578303] (req 20574208->20578303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760732.287718:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20574208 -> 20578304 00010000:00010000:1.0:1789760732.287739:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20574208->20578303] (req 20574208->20578303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.287751:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20574208->20578303] (req 20574208->20578303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.287761:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20574208->20578303] (req 20574208->20578303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.287768:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760732.287773:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20574208->20578303] (req 20574208->20578303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.287782:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20574208->20578303] (req 20574208->20578303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.287800:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20574208->20578303] (req 20574208->20578303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.287809:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208316 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20574208->20578303] (req 20574208->20578303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760732.287903:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484206464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33132us (33360us total) trans 4294972341 rc 0/0 00000100:00100000:1.0:1789760732.287914:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5052 00000100:00100000:1.0:1789760732.287922:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721521861483 time 721585685489 state 0x2 ref 0 00000001:00080000:3.0:1789760732.287995:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972341 is committed 00000100:00100000:1.0:1789760732.291802:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760732.291892:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484206592 00000100:00100000:0.0:1789760732.291901:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484206592 00000100:00100000:0.0:1789760732.291961:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721521861483 time 721585685489 state 0x0 ref 1 00000100:00100000:0.0:1789760732.291977:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721585945905 time 721585685489 state 0x1 ref 1 00000100:00100000:0.0:1789760732.292009:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721585945905 time 721622939515 state 0x0 ref 1 00000100:00100000:0.0:1789760732.292022:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484206592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760732.292114:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20578304, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:0.0:1789760732.292124:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.292147:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:0.0:1789760732.292152:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760732.292158:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20578304->20582399] (req 20578304->20582399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.292179:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20578304->20582399] (req 20578304->20582399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760732.327180:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972342 is committed 00002000:00010000:2.0:1789760732.327338:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20578304 -> 20582400 00010000:00010000:2.0:1789760732.327354:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20578304->20582399] (req 20578304->20582399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.327367:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20578304->20582399] (req 20578304->20582399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.327376:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20578304->20582399] (req 20578304->20582399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.327399:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760732.327405:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20578304->20582399] (req 20578304->20582399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.327414:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20578304->20582399] (req 20578304->20582399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.327542:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20578304->20582399] (req 20578304->20582399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.327552:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620831d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20578304->20582399] (req 20578304->20582399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760732.327644:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484206592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35620us (35848us total) trans 4294972342 rc 0/0 00000100:00100000:2.0:1789760732.327653:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5053 00000100:00100000:2.0:1789760732.327663:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721585945905 time 721622939515 state 0x2 ref 0 00000100:00100000:2.0:1789760732.336723:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760732.336812:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484206720 00000100:00100000:3.0:1789760732.336883:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484206720 00000100:00100000:3.0:1789760732.336935:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721585945905 time 721622939515 state 0x0 ref 1 00000100:00100000:3.0:1789760732.336950:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721623199931 time 721622939515 state 0x1 ref 1 00000100:00100000:3.0:1789760732.336980:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721623199931 time 721667911014 state 0x0 ref 1 00000100:00100000:3.0:1789760732.336991:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484206720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760732.337036:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20582400, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:3.0:1789760732.337045:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.337060:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:3.0:1789760732.337167:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760732.337175:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20582400->20586495] (req 20582400->20586495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.337189:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20582400->20586495] (req 20582400->20586495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760732.389675:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760732.389707:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760732.389730:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760732.389746:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760732.389765:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760732.389788:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484206976 00000100:00100000:3.0:1789760732.389797:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484206976 00000100:00100000:3.0:1789760732.389858:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484207232 00000100:00100000:3.0:1789760732.389861:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484207232 00000100:00100000:3.0:1789760732.389877:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484207104 00000100:00100000:3.0:1789760732.389880:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484207104 00000100:00100000:3.0:1789760732.389890:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484207360 00000100:00100000:3.0:1789760732.389892:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484207360 00000100:00100000:3.0:1789760732.389905:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 901 00000100:00100000:3.0:1789760732.389913:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2422:x1876699484206976:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760732.389977:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2422:x1876699484206976:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 64us (316us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760732.389985:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 901 00000100:00100000:3.0:1789760732.390000:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 902 00000100:00100000:3.0:1789760732.390006:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484207232:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760732.390206:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484207232:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 199us (497us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760732.390212:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 902 00000100:00100000:3.0:1789760732.390227:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 903 00000100:00100000:3.0:1789760732.390234:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484207104:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760732.390258:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484207104:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 25us (510us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760732.390264:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 903 00000100:00100000:3.0:1789760732.390275:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 904 00000100:00100000:3.0:1789760732.390280:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5081180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484207360:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760732.390300:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5081180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484207360:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 20us (535us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760732.390307:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 904 00000100:00100000:3.0:1789760732.390326:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484206848 00000100:00100000:3.0:1789760732.390329:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484206848 00000100:00100000:3.0:1789760732.390351:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 475 00000100:00100000:3.0:1789760732.390357:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b5850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484206848:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760732.390377:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b5850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484206848:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 20us (647us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760732.390393:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 475 00000100:00080000:3.0:1789760732.390421:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:3.0:1789760732.390425:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000001:00080000:3.0:1789760732.394554:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972343 is committed 00002000:00010000:1.0:1789760732.395963:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20582400 -> 20586496 00010000:00010000:1.0:1789760732.395982:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20582400->20586495] (req 20582400->20586495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.395996:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20582400->20586495] (req 20582400->20586495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.396008:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20582400->20586495] (req 20582400->20586495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.396016:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760732.396021:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20582400->20586495] (req 20582400->20586495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.396030:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20582400->20586495] (req 20582400->20586495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.396052:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20582400->20586495] (req 20582400->20586495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.397727:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208324 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20582400->20586495] (req 20582400->20586495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760732.397947:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484206720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60952us (61229us total) trans 4294972343 rc 0/0 00000100:00100000:1.0:1789760732.397958:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5054 00000100:00100000:1.0:1789760732.397969:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721623199931 time 721667911014 state 0x2 ref 0 00000100:00100000:3.0:1789760732.404190:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760732.406472:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484207488 00000100:00100000:1.0:1789760732.406485:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484207488 00000100:00100000:1.0:1789760732.406544:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721623199931 time 721667911014 state 0x0 ref 1 00000100:00100000:1.0:1789760732.406559:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721668171430 time 721667911014 state 0x1 ref 1 00000100:00100000:1.0:1789760732.406597:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721668171430 time 721737528335 state 0x0 ref 1 00000100:00100000:1.0:1789760732.406610:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484207488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760732.406658:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20586496, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:1.0:1789760732.406667:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.406683:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:1.0:1789760732.406688:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760732.406694:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20586496->20590591] (req 20586496->20590591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.406712:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20586496->20590591] (req 20586496->20590591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760732.442969:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972344 is committed 00002000:00010000:1.0:1789760732.443087:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20586496 -> 20590592 00010000:00010000:1.0:1789760732.443102:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20586496->20590591] (req 20586496->20590591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.443114:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20586496->20590591] (req 20586496->20590591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.443123:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20586496->20590591] (req 20586496->20590591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.443130:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760732.443135:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20586496->20590591] (req 20586496->20590591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.443143:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20586496->20590591] (req 20586496->20590591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.443161:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20586496->20590591] (req 20586496->20590591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.443179:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620832b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20586496->20590591] (req 20586496->20590591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760732.443307:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484207488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36695us (39122us total) trans 4294972344 rc 0/0 00000100:00100000:1.0:1789760732.443318:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5055 00000100:00100000:1.0:1789760732.443329:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721668171430 time 721737528335 state 0x2 ref 0 00000100:00100000:2.0:1789760732.449000:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760732.449275:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484207616 00000100:00100000:1.0:1789760732.449288:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484207616 00000100:00100000:1.0:1789760732.449363:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721668171430 time 721737528335 state 0x0 ref 1 00000100:00100000:1.0:1789760732.449391:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721737788751 time 721737528335 state 0x1 ref 1 00000100:00100000:1.0:1789760732.449427:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721737788751 time 721780357341 state 0x0 ref 1 00000100:00100000:1.0:1789760732.449440:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484207616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760732.449619:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20590592, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:1.0:1789760732.449631:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.449652:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:1.0:1789760732.449658:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760732.449665:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20590592->20594687] (req 20590592->20594687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.449682:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20590592->20594687] (req 20590592->20594687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760732.499860:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20590592 -> 20594688 00010000:00010000:1.0:1789760732.499887:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20590592->20594687] (req 20590592->20594687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.499904:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20590592->20594687] (req 20590592->20594687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.499915:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20590592->20594687] (req 20590592->20594687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.499922:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760732.499928:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20590592->20594687] (req 20590592->20594687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.499937:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20590592->20594687] (req 20590592->20594687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.499962:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20590592->20594687] (req 20590592->20594687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.499970:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208332 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20590592->20594687] (req 20590592->20594687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760732.500142:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484207616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50697us (51150us total) trans 4294972345 rc 0/0 00000100:00100000:1.0:1789760732.500158:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5056 00000100:00100000:1.0:1789760732.500237:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721737788751 time 721780357341 state 0x2 ref 0 00000001:00080000:0.0:1789760732.501429:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972345 is committed 00000100:00100000:2.0:1789760732.503312:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760732.503466:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484207744 00000100:00100000:0.0:1789760732.503489:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484207744 00000100:00100000:0.0:1789760732.503572:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721737788751 time 721780357341 state 0x0 ref 1 00000100:00100000:0.0:1789760732.503594:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721780617757 time 721780357341 state 0x1 ref 1 00000100:00100000:0.0:1789760732.503625:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721780617757 time 721834556133 state 0x0 ref 1 00000100:00100000:0.0:1789760732.503641:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484207744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760732.503706:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20594688, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:0.0:1789760732.503717:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.503734:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:0.0:1789760732.503739:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760732.503746:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20594688->20598783] (req 20594688->20598783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.503763:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20594688->20598783] (req 20594688->20598783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760732.540316:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20594688 -> 20598784 00010000:00010000:0.0:1789760732.540339:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20594688->20598783] (req 20594688->20598783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.540351:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20594688->20598783] (req 20594688->20598783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.540358:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20594688->20598783] (req 20594688->20598783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.540362:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760732.540366:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20594688->20598783] (req 20594688->20598783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.540375:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20594688->20598783] (req 20594688->20598783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.540409:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20594688->20598783] (req 20594688->20598783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.540426:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208339 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20594688->20598783] (req 20594688->20598783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760732.540489:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972346 is committed 00000100:00100000:0.0:1789760732.540567:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484207744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36923us (37263us total) trans 4294972346 rc 0/0 00000100:00100000:0.0:1789760732.540581:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5057 00000100:00100000:0.0:1789760732.540593:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721780617757 time 721834556133 state 0x2 ref 0 00000100:00100000:1.0:1789760732.545015:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760732.545150:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484207872 00000100:00100000:2.0:1789760732.545166:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484207872 00000100:00100000:2.0:1789760732.545239:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721780617757 time 721834556133 state 0x0 ref 1 00000100:00100000:2.0:1789760732.545260:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721834816549 time 721834556133 state 0x1 ref 1 00000100:00100000:2.0:1789760732.545304:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721834816549 time 721876235016 state 0x0 ref 1 00000100:00100000:2.0:1789760732.545318:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484207872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760732.545405:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20598784, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:2.0:1789760732.545416:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.545436:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:2.0:1789760732.545441:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760732.545450:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20598784->20602879] (req 20598784->20602879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.545477:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20598784->20602879] (req 20598784->20602879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760732.591780:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20598784 -> 20602880 00010000:00010000:2.0:1789760732.591804:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20598784->20602879] (req 20598784->20602879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.591877:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20598784->20602879] (req 20598784->20602879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.591887:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20598784->20602879] (req 20598784->20602879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.591901:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760732.591907:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20598784->20602879] (req 20598784->20602879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.591917:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20598784->20602879] (req 20598784->20602879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.591941:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20598784->20602879] (req 20598784->20602879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.591949:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208340 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20598784->20602879] (req 20598784->20602879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760732.592097:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484207872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46741us (47057us total) trans 4294972347 rc 0/0 00000100:00100000:2.0:1789760732.592110:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5058 00000100:00100000:2.0:1789760732.592121:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721834816549 time 721876235016 state 0x2 ref 0 00000001:00080000:3.0:1789760732.592895:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972347 is committed 00000100:00100000:2.0:1789760732.595680:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760732.595777:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484208000 00000100:00100000:1.0:1789760732.595791:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484208000 00000100:00100000:1.0:1789760732.595874:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721834816549 time 721876235016 state 0x0 ref 1 00000100:00100000:1.0:1789760732.595894:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721876495432 time 721876235016 state 0x1 ref 1 00000100:00100000:1.0:1789760732.595929:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721876495432 time 721926859551 state 0x0 ref 1 00000100:00100000:1.0:1789760732.595949:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484208000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760732.596017:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20602880, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:1.0:1789760732.596028:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.596047:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=721s v=(0 0) 00010000:00010000:1.0:1789760732.596052:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760732.596060:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20602880->20606975] (req 20602880->20606975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.596112:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20602880->20606975] (req 20602880->20606975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760732.687699:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972348 is committed 00002000:00010000:2.0:1789760732.687943:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20602880 -> 20606976 00010000:00010000:2.0:1789760732.687965:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20602880->20606975] (req 20602880->20606975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.687982:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20602880->20606975] (req 20602880->20606975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.688003:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20602880->20606975] (req 20602880->20606975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.688012:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760732.688019:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20602880->20606975] (req 20602880->20606975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.688032:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20602880->20606975] (req 20602880->20606975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.688097:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20602880->20606975] (req 20602880->20606975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.688106:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208347 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20602880->20606975] (req 20602880->20606975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760732.688233:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484208000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 92285us (92561us total) trans 4294972348 rc 0/0 00000100:00100000:2.0:1789760732.688245:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5059 00000100:00100000:2.0:1789760732.688257:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721876495432 time 721926859551 state 0x2 ref 0 00000100:00100000:0.0:1789760732.691745:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760732.694577:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484208128 00000100:00100000:3.0:1789760732.694600:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484208128 00000100:00100000:2.0:1789760732.694952:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721876495432 time 721926859551 state 0x0 ref 1 00000100:00100000:2.0:1789760732.694973:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721927119967 time 721926859551 state 0x1 ref 1 00000100:00100000:2.0:1789760732.695020:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721927119967 time 722025950222 state 0x0 ref 1 00000100:00100000:2.0:1789760732.695036:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484208128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760732.695135:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20606976, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:2.0:1789760732.695146:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.695167:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:2.0:1789760732.695172:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760732.695179:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20606976->20611071] (req 20606976->20611071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.695196:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20606976->20611071] (req 20606976->20611071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760732.750709:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972349 is committed 00002000:00010000:2.0:1789760732.750800:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20606976 -> 20611072 00010000:00010000:2.0:1789760732.750822:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20606976->20611071] (req 20606976->20611071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.750838:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20606976->20611071] (req 20606976->20611071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.750849:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20606976->20611071] (req 20606976->20611071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.750858:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760732.750865:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20606976->20611071] (req 20606976->20611071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.750876:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20606976->20611071] (req 20606976->20611071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.750901:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20606976->20611071] (req 20606976->20611071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.750910:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620834e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20606976->20611071] (req 20606976->20611071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760732.751184:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484208128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56144us (59447us total) trans 4294972349 rc 0/0 00000100:00100000:2.0:1789760732.751198:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5060 00000100:00100000:2.0:1789760732.751211:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721927119967 time 722025950222 state 0x2 ref 0 00000100:00100000:3.0:1789760732.753592:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760732.753707:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484208256 00000100:00100000:0.0:1789760732.753720:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484208256 00000100:00100000:0.0:1789760732.753800:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 721927119967 time 722025950222 state 0x0 ref 1 00000100:00100000:0.0:1789760732.753818:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722026210638 time 722025950222 state 0x1 ref 1 00000100:00100000:0.0:1789760732.753895:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722026210638 time 722084824362 state 0x0 ref 1 00000100:00100000:0.0:1789760732.753912:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484208256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760732.753978:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20611072, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:0.0:1789760732.753989:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.754009:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:0.0:1789760732.754014:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760732.754022:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20611072->20615167] (req 20611072->20615167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.754039:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20611072->20615167] (req 20611072->20615167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760732.792730:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20611072 -> 20615168 00010000:00010000:2.0:1789760732.792757:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20611072->20615167] (req 20611072->20615167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.792773:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20611072->20615167] (req 20611072->20615167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.792783:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20611072->20615167] (req 20611072->20615167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.792798:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760732.792805:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20611072->20615167] (req 20611072->20615167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.792816:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20611072->20615167] (req 20611072->20615167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.792840:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20611072->20615167] (req 20611072->20615167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.792850:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208355 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20611072->20615167] (req 20611072->20615167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760732.792990:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484208256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39075us (39413us total) trans 4294972350 rc 0/0 00000100:00100000:2.0:1789760732.793002:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5061 00000100:00100000:2.0:1789760732.793015:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722026210638 time 722084824362 state 0x2 ref 0 00000001:00080000:3.0:1789760732.793111:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972350 is committed 00000100:00100000:3.0:1789760732.798188:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760732.798275:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484208384 00000100:00100000:1.0:1789760732.798289:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484208384 00000100:00100000:1.0:1789760732.798362:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722026210638 time 722084824362 state 0x0 ref 1 00000100:00100000:1.0:1789760732.798390:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722085084778 time 722084824362 state 0x1 ref 1 00000100:00100000:1.0:1789760732.798458:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722085084778 time 722129387648 state 0x0 ref 1 00000100:00100000:1.0:1789760732.798472:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484208384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760732.798534:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20615168, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:1.0:1789760732.798544:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.798580:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:1.0:1789760732.798585:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760732.798593:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20615168->20619263] (req 20615168->20619263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760732.798610:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20615168->20619263] (req 20615168->20619263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760732.846819:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20615168 -> 20619264 00000001:00080000:0.0:1789760732.846834:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972351 is committed 00010000:00010000:2.0:1789760732.846849:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20615168->20619263] (req 20615168->20619263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.846866:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20615168->20619263] (req 20615168->20619263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.846882:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20615168->20619263] (req 20615168->20619263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.846891:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760732.846897:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20615168->20619263] (req 20615168->20619263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.846906:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20615168->20619263] (req 20615168->20619263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.846930:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20615168->20619263] (req 20615168->20619263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.846939:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620835c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20615168->20619263] (req 20615168->20619263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760732.847063:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484208384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48587us (48883us total) trans 4294972351 rc 0/0 00000100:00100000:2.0:1789760732.847114:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5062 00000100:00100000:2.0:1789760732.847127:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722085084778 time 722129387648 state 0x2 ref 0 00000100:00100000:0.0:1789760732.852228:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760732.852606:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484208512 00000100:00100000:2.0:1789760732.852621:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484208512 00000100:00100000:2.0:1789760732.852693:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722085084778 time 722129387648 state 0x0 ref 1 00000100:00100000:2.0:1789760732.852712:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722129648064 time 722129387648 state 0x1 ref 1 00000100:00100000:2.0:1789760732.852764:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722129648064 time 722183683254 state 0x0 ref 1 00000100:00100000:2.0:1789760732.852779:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484208512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760732.852842:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20619264, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:2.0:1789760732.852853:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.852874:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:2.0:1789760732.852880:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760732.852889:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20619264->20623359] (req 20619264->20623359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.852907:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20619264->20623359] (req 20619264->20623359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760732.920011:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20619264 -> 20623360 00010000:00010000:2.0:1789760732.920038:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20619264->20623359] (req 20619264->20623359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.920054:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20619264->20623359] (req 20619264->20623359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.920082:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20619264->20623359] (req 20619264->20623359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.920090:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760732.920096:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20619264->20623359] (req 20619264->20623359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.920107:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20619264->20623359] (req 20619264->20623359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.920130:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20619264->20623359] (req 20619264->20623359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.920137:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208363 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20619264->20623359] (req 20619264->20623359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760732.920273:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484208512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67489us (68051us total) trans 4294972352 rc 0/0 00000100:00100000:2.0:1789760732.920286:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5063 00000100:00100000:2.0:1789760732.920298:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722129648064 time 722183683254 state 0x2 ref 0 00000001:00080000:3.0:1789760732.921113:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972352 is committed 00000100:00100000:3.0:1789760732.924322:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760732.924488:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484208640 00000100:00100000:2.0:1789760732.924502:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484208640 00000100:00100000:2.0:1789760732.924576:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722129648064 time 722183683254 state 0x0 ref 1 00000100:00100000:2.0:1789760732.924595:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722183943670 time 722183683254 state 0x1 ref 1 00000100:00100000:2.0:1789760732.924638:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722183943670 time 722255568441 state 0x0 ref 1 00000100:00100000:2.0:1789760732.924651:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484208640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760732.924720:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20623360, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:2.0:1789760732.924731:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.924751:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:2.0:1789760732.924756:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760732.924763:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20623360->20627455] (req 20623360->20627455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760732.924778:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20623360->20627455] (req 20623360->20627455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760732.975742:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972353 is committed 00002000:00010000:3.0:1789760732.976804:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20623360 -> 20627456 00010000:00010000:3.0:1789760732.976828:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20623360->20627455] (req 20623360->20627455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.976851:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20623360->20627455] (req 20623360->20627455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.976861:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20623360->20627455] (req 20623360->20627455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.976870:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760732.976875:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20623360->20627455] (req 20623360->20627455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.976885:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20623360->20627455] (req 20623360->20627455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.976909:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20623360->20627455] (req 20623360->20627455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760732.976916:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620836a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20623360->20627455] (req 20623360->20627455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760732.977507:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484208640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52848us (53189us total) trans 4294972353 rc 0/0 00000100:00100000:3.0:1789760732.977522:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5064 00000100:00100000:3.0:1789760732.977535:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722183943670 time 722255568441 state 0x2 ref 0 00000100:00100000:1.0:1789760732.983360:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760732.983514:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484208768 00000100:00100000:0.0:1789760732.983528:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484208768 00000100:00100000:0.0:1789760732.983616:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722183943670 time 722255568441 state 0x0 ref 1 00000100:00100000:0.0:1789760732.983638:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722255828857 time 722255568441 state 0x1 ref 1 00000100:00100000:0.0:1789760732.983722:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722255828857 time 722314652293 state 0x0 ref 1 00000100:00100000:0.0:1789760732.983738:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484208768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760732.983800:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20627456, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:0.0:1789760732.983810:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.983830:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:0.0:1789760732.983834:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760732.983841:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20627456->20631551] (req 20627456->20631551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760732.983857:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20627456->20631551] (req 20627456->20631551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760733.013359:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20627456 -> 20631552 00010000:00010000:2.0:1789760733.013396:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20627456->20631551] (req 20627456->20631551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.013412:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20627456->20631551] (req 20627456->20631551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.013422:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20627456->20631551] (req 20627456->20631551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.013430:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760733.013436:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20627456->20631551] (req 20627456->20631551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.013447:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20627456->20631551] (req 20627456->20631551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.013477:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20627456->20631551] (req 20627456->20631551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.013485:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208371 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20627456->20631551] (req 20627456->20631551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760733.013619:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484208768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29877us (30267us total) trans 4294972354 rc 0/0 00000100:00100000:2.0:1789760733.013633:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5065 00000100:00100000:2.0:1789760733.013646:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722255828857 time 722314652293 state 0x2 ref 0 00000001:00080000:3.0:1789760733.013738:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972354 is committed 00000100:00100000:0.0:1789760733.022042:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760733.025127:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484208896 00000100:00100000:1.0:1789760733.025269:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484208896 00000100:00100000:1.0:1789760733.025359:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722255828857 time 722314652293 state 0x0 ref 1 00000100:00100000:1.0:1789760733.025388:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722314912709 time 722314652293 state 0x1 ref 1 00000100:00100000:1.0:1789760733.025499:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722314912709 time 722356427575 state 0x0 ref 1 00000100:00100000:1.0:1789760733.025516:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484208896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760733.025584:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20631552, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:1.0:1789760733.025595:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.025614:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:1.0:1789760733.025619:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760733.025628:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20631552->20635647] (req 20631552->20635647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.025644:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20631552->20635647] (req 20631552->20635647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760733.068329:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508917248:0@lo:13:osp-pre-0-1.0 00000100:00100000:1.0:1789760733.068449:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501b80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508917504:0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760733.068449:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760733.068532:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1789760733.068540:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df82c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508917376:0@lo:13:osp-pre-0-0.0 00000100:00100000:1.0:1789760733.068558:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760733.068574:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760733.068615:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508917504 00010000:00010000:3.0:1789760733.068622:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760733.068626:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508917504 00000100:00100000:3.0:1789760733.068639:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508917248 00000100:00100000:3.0:1789760733.068645:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508917248 00000100:00100000:1.0:1789760733.068686:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 433 00000100:00100000:1.0:1789760733.068699:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508917504:12345-0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760733.068707:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508917376 00000100:00100000:3.0:1789760733.068712:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508917376 00000100:00100000:3.0:1789760733.068740:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 374 00000100:00100000:3.0:1789760733.068749:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508917248:12345-0@lo:13:osp-pre-0-1.0 00000004:00000024:1.0:1789760733.068751:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00002000:00000024:3.0:1789760733.068768:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527892480 pending 1703936 free 3837788160 avail 3805282304 00000100:00100000:1.0:1789760733.068816:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508917504:12345-0@lo:41:osp-pre-0-1.0 Request processed in 119us (258us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760733.068824:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508917248:12345-0@lo:13:osp-pre-0-1.0 Request processed in 75us (376us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760733.068829:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 433 00000100:00100000:3.0:1789760733.068839:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 374 00000100:00100000:3.0:1789760733.068857:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 375 00000100:00100000:3.0:1789760733.068865:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508917376:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:3.0:1789760733.068876:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527892480 pending 1703936 free 3837788160 avail 3805282304 00000100:00100000:3.0:1789760733.068902:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508917376:12345-0@lo:13:osp-pre-0-0.0 Request processed in 39us (330us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760733.068908:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 375 00000100:00080000:3.0:1789760733.068935:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760733.068980:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df82c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508917376:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760733.068994:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508917248:0@lo:13:osp-pre-0-1.0 00000100:00100000:1.0:1789760733.072579:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501b80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508917504:0@lo:41:osp-pre-0-1.0 00000001:00080000:3.0:1789760733.089900:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972355 is committed 00002000:00010000:0.0:1789760733.090139:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20631552 -> 20635648 00010000:00010000:0.0:1789760733.090159:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20631552->20635647] (req 20631552->20635647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.090174:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20631552->20635647] (req 20631552->20635647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.090184:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20631552->20635647] (req 20631552->20635647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.090192:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760733.090198:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20631552->20635647] (req 20631552->20635647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.090208:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20631552->20635647] (req 20631552->20635647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.090232:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20631552->20635647] (req 20631552->20635647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.090240:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208378 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20631552->20635647] (req 20631552->20635647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760733.094460:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484208896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68928us (72413us total) trans 4294972355 rc 0/0 00000100:00100000:0.0:1789760733.094481:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5066 00000100:00100000:0.0:1789760733.094496:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722314912709 time 722356427575 state 0x2 ref 0 00000100:00100000:0.0:1789760733.098473:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760733.098641:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484209024 00000100:00100000:3.0:1789760733.098663:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484209024 00000100:00100000:3.0:1789760733.098733:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722314912709 time 722356427575 state 0x0 ref 1 00000100:00100000:3.0:1789760733.098753:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722356687991 time 722356427575 state 0x1 ref 1 00000100:00100000:3.0:1789760733.098785:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722356687991 time 722429715276 state 0x0 ref 1 00000100:00100000:3.0:1789760733.098799:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484209024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760733.098865:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20635648, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:3.0:1789760733.098876:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.098896:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:3.0:1789760733.098901:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760733.098909:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20635648->20639743] (req 20635648->20639743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.098927:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20635648->20639743] (req 20635648->20639743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760733.183103:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972356 is committed 00002000:00010000:1.0:1789760733.183361:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20635648 -> 20639744 00010000:00010000:1.0:1789760733.183391:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20635648->20639743] (req 20635648->20639743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.183409:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20635648->20639743] (req 20635648->20639743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.183419:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20635648->20639743] (req 20635648->20639743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.183427:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.183434:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20635648->20639743] (req 20635648->20639743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.183443:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20635648->20639743] (req 20635648->20639743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.183468:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20635648->20639743] (req 20635648->20639743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.183477:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620837f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20635648->20639743] (req 20635648->20639743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.183606:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484209024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 84802us (85143us total) trans 4294972356 rc 0/0 00000100:00100000:1.0:1789760733.183618:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5067 00000100:00100000:1.0:1789760733.183631:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722356687991 time 722429715276 state 0x2 ref 0 00000100:00100000:1.0:1789760733.186637:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760733.188474:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484209152 00000100:00100000:3.0:1789760733.188497:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484209152 00000100:00100000:3.0:1789760733.188587:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722356687991 time 722429715276 state 0x0 ref 1 00000100:00100000:3.0:1789760733.188609:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722429975692 time 722429715276 state 0x1 ref 1 00000100:00100000:3.0:1789760733.188769:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722429975692 time 722519697683 state 0x0 ref 1 00000100:00100000:3.0:1789760733.188785:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484209152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760733.188856:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20639744, mtime: 0x6aad94dc, atime: 0x0, ctime: 0x6aad94dc, blocks: 0xa007 00010000:00010000:3.0:1789760733.188867:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.188886:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:3.0:1789760733.188891:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760733.188898:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20639744->20643839] (req 20639744->20643839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.188915:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20639744->20643839] (req 20639744->20643839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760733.276979:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972357 is committed 00002000:00010000:1.0:1789760733.277131:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20639744 -> 20643840 00002000:00010000:1.0:1789760733.277137:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760732 -> 1789760733 00002000:00010000:1.0:1789760733.277140:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760732 -> 1789760733 00010000:00010000:1.0:1789760733.277157:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20639744->20643839] (req 20639744->20643839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.277182:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20639744->20643839] (req 20639744->20643839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.277193:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20639744->20643839] (req 20639744->20643839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.277202:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.277207:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20639744->20643839] (req 20639744->20643839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.277219:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20639744->20643839] (req 20639744->20643839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.277244:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20639744->20643839] (req 20639744->20643839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.277252:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208386 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20639744->20643839] (req 20639744->20643839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.277368:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484209152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 88581us (90740us total) trans 4294972357 rc 0/0 00000100:00100000:1.0:1789760733.277389:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5068 00000100:00100000:1.0:1789760733.277402:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722429975692 time 722519697683 state 0x2 ref 0 00000100:00100000:2.0:1789760733.280857:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760733.281144:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484209280 00000100:00100000:3.0:1789760733.281220:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484209280 00000100:00100000:3.0:1789760733.281300:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722429975692 time 722519697683 state 0x0 ref 1 00000100:00100000:3.0:1789760733.281320:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722519958099 time 722519697683 state 0x1 ref 1 00000100:00100000:3.0:1789760733.281397:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722519958099 time 722612327522 state 0x0 ref 1 00000100:00100000:3.0:1789760733.281412:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484209280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760733.281495:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20643840, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:3.0:1789760733.281506:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.281529:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:3.0:1789760733.281534:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760733.281542:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20643840->20647935] (req 20643840->20647935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.281559:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20643840->20647935] (req 20643840->20647935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760733.319966:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972358 is committed 00002000:00010000:3.0:1789760733.320148:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20643840 -> 20647936 00010000:00010000:3.0:1789760733.320236:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20643840->20647935] (req 20643840->20647935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.320252:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20643840->20647935] (req 20643840->20647935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.320263:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20643840->20647935] (req 20643840->20647935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.320272:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760733.320277:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20643840->20647935] (req 20643840->20647935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.320288:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20643840->20647935] (req 20643840->20647935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.320321:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20643840->20647935] (req 20643840->20647935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.320330:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620838d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20643840->20647935] (req 20643840->20647935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760733.320585:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484209280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39169us (39736us total) trans 4294972358 rc 0/0 00000100:00100000:3.0:1789760733.320598:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5069 00000100:00100000:3.0:1789760733.320610:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722519958099 time 722612327522 state 0x2 ref 0 00000100:00100000:0.0:1789760733.325045:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760733.325182:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484209408 00000100:00100000:2.0:1789760733.325197:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484209408 00000100:00100000:2.0:1789760733.325268:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722519958099 time 722612327522 state 0x0 ref 1 00000100:00100000:2.0:1789760733.325286:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722612587938 time 722612327522 state 0x1 ref 1 00000100:00100000:2.0:1789760733.325316:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722612587938 time 722656246920 state 0x0 ref 1 00000100:00100000:2.0:1789760733.325328:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484209408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760733.325404:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20647936, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:2.0:1789760733.325415:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.325438:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:2.0:1789760733.325444:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760733.325451:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20647936->20652031] (req 20647936->20652031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.325476:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20647936->20652031] (req 20647936->20652031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760733.367127:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972359 is committed 00002000:00010000:1.0:1789760733.367389:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20647936 -> 20652032 00010000:00010000:1.0:1789760733.367409:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20647936->20652031] (req 20647936->20652031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.367425:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20647936->20652031] (req 20647936->20652031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.367435:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20647936->20652031] (req 20647936->20652031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.367443:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.367448:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20647936->20652031] (req 20647936->20652031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.367459:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20647936->20652031] (req 20647936->20652031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.367552:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20647936->20652031] (req 20647936->20652031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.367567:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208394 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20647936->20652031] (req 20647936->20652031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.367858:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484209408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42521us (42819us total) trans 4294972359 rc 0/0 00000100:00100000:1.0:1789760733.367871:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5070 00000100:00100000:1.0:1789760733.367885:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722612587938 time 722656246920 state 0x2 ref 0 00000100:00100000:0.0:1789760733.371252:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760733.371407:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484209536 00000100:00100000:1.0:1789760733.371422:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484209536 00000100:00100000:1.0:1789760733.371569:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722612587938 time 722656246920 state 0x0 ref 1 00000100:00100000:1.0:1789760733.371589:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722656507336 time 722656246920 state 0x1 ref 1 00000100:00100000:1.0:1789760733.371681:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722656507336 time 722702610296 state 0x0 ref 1 00000100:00100000:1.0:1789760733.371697:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484209536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760733.371768:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20652032, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:1.0:1789760733.371779:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.371800:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:1.0:1789760733.371805:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760733.371813:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20652032->20656127] (req 20652032->20656127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.371835:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20652032->20656127] (req 20652032->20656127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760733.417437:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972360 is committed 00002000:00010000:1.0:1789760733.417690:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20652032 -> 20656128 00010000:00010000:1.0:1789760733.417712:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20652032->20656127] (req 20652032->20656127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.417728:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20652032->20656127] (req 20652032->20656127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.417739:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20652032->20656127] (req 20652032->20656127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.417748:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.417754:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20652032->20656127] (req 20652032->20656127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.417766:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20652032->20656127] (req 20652032->20656127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.417792:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20652032->20656127] (req 20652032->20656127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.417801:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620839b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20652032->20656127] (req 20652032->20656127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.417951:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484209536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46252us (46709us total) trans 4294972360 rc 0/0 00000100:00100000:1.0:1789760733.417964:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5071 00000100:00100000:1.0:1789760733.417977:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722656507336 time 722702610296 state 0x2 ref 0 00000100:00100000:2.0:1789760733.421765:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760733.421916:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484209664 00000100:00100000:3.0:1789760733.421931:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484209664 00000100:00100000:3.0:1789760733.422004:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722656507336 time 722702610296 state 0x0 ref 1 00000100:00100000:3.0:1789760733.422023:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722702870712 time 722702610296 state 0x1 ref 1 00000100:00100000:3.0:1789760733.422061:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722702870712 time 722752991575 state 0x0 ref 1 00000100:00100000:3.0:1789760733.422103:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484209664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760733.422291:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20656128, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:3.0:1789760733.422302:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.422323:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:3.0:1789760733.422329:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760733.422337:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20656128->20660223] (req 20656128->20660223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.422353:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20656128->20660223] (req 20656128->20660223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760733.454925:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20656128 -> 20660224 00010000:00010000:3.0:1789760733.454952:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20656128->20660223] (req 20656128->20660223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.454967:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20656128->20660223] (req 20656128->20660223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.454978:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20656128->20660223] (req 20656128->20660223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.454986:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760733.454991:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20656128->20660223] (req 20656128->20660223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.455001:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20656128->20660223] (req 20656128->20660223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.455027:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20656128->20660223] (req 20656128->20660223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.455035:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083a2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20656128->20660223] (req 20656128->20660223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760733.455208:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484209664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33101us (33452us total) trans 4294972361 rc 0/0 00000100:00100000:3.0:1789760733.455221:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5072 00000100:00100000:3.0:1789760733.455233:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722702870712 time 722752991575 state 0x2 ref 0 00000001:00080000:3.0:1789760733.455281:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972361 is committed 00000100:00100000:0.0:1789760733.458839:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760733.458948:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484209792 00000100:00100000:3.0:1789760733.458962:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484209792 00000100:00100000:3.0:1789760733.459035:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722702870712 time 722752991575 state 0x0 ref 1 00000100:00100000:3.0:1789760733.459054:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722753251991 time 722752991575 state 0x1 ref 1 00000100:00100000:3.0:1789760733.459134:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722753251991 time 722790064453 state 0x0 ref 1 00000100:00100000:3.0:1789760733.459149:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484209792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760733.459284:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20660224, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:3.0:1789760733.459293:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.459312:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:3.0:1789760733.459317:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760733.459324:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20660224->20664319] (req 20660224->20664319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.459341:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20660224->20664319] (req 20660224->20664319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760733.523596:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972362 is committed 00002000:00010000:1.0:1789760733.523688:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20660224 -> 20664320 00010000:00010000:1.0:1789760733.523709:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20660224->20664319] (req 20660224->20664319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.523724:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20660224->20664319] (req 20660224->20664319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.523735:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20660224->20664319] (req 20660224->20664319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.523743:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.523748:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20660224->20664319] (req 20660224->20664319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.523758:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20660224->20664319] (req 20660224->20664319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.523782:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20660224->20664319] (req 20660224->20664319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.523794:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962083a9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20660224->20664319] (req 20660224->20664319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.523913:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484209792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 64761us (65084us total) trans 4294972362 rc 0/0 00000100:00100000:1.0:1789760733.523927:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5073 00000100:00100000:1.0:1789760733.523940:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722753251991 time 722790064453 state 0x2 ref 0 00000100:00100000:3.0:1789760733.538441:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760733.538752:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484209920 00000100:00100000:1.0:1789760733.538765:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484209920 00000100:00100000:1.0:1789760733.538837:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722753251991 time 722790064453 state 0x0 ref 1 00000100:00100000:1.0:1789760733.538856:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722790324869 time 722790064453 state 0x1 ref 1 00000100:00100000:1.0:1789760733.538889:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722790324869 time 722869819893 state 0x0 ref 1 00000100:00100000:1.0:1789760733.538902:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484209920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760733.538980:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20664320, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:1.0:1789760733.538990:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.539012:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:1.0:1789760733.539017:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760733.539025:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20664320->20668415] (req 20664320->20668415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.539047:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20664320->20668415] (req 20664320->20668415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760733.581694:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f45880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508917632:0@lo:41:osp-pre-1-0.0 00000100:00100000:2.0:1789760733.581795:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760733.582022:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508917632 00000100:00100000:1.0:1789760733.582033:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508917632 00000100:00100000:1.0:1789760733.582127:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 434 00000100:00100000:1.0:1789760733.582139:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508917632:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760733.582199:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760733.582257:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508917632:12345-0@lo:41:osp-pre-1-0.0 Request processed in 120us (462us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760733.582266:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 434 00000100:00100000:0.0:1789760733.583136:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f45880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508917632:0@lo:41:osp-pre-1-0.0 00000001:00080000:3.0:1789760733.585152:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972363 is committed 00002000:00010000:1.0:1789760733.585355:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20664320 -> 20668416 00010000:00010000:1.0:1789760733.585383:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20664320->20668415] (req 20664320->20668415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.585400:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20664320->20668415] (req 20664320->20668415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.585410:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20664320->20668415] (req 20664320->20668415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.585417:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.585423:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20664320->20668415] (req 20664320->20668415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.585438:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20664320->20668415] (req 20664320->20668415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.585610:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20664320->20668415] (req 20664320->20668415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.585621:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083b0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20664320->20668415] (req 20664320->20668415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.585738:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484209920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46831us (47304us total) trans 4294972363 rc 0/0 00000100:00100000:1.0:1789760733.585750:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5074 00000100:00100000:1.0:1789760733.585763:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722790324869 time 722869819893 state 0x2 ref 0 00000100:00100000:0.0:1789760733.588411:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760733.588495:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484210048 00000100:00100000:1.0:1789760733.588508:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484210048 00000100:00100000:1.0:1789760733.588583:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722790324869 time 722869819893 state 0x0 ref 1 00000100:00100000:1.0:1789760733.588602:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722870080309 time 722869819893 state 0x1 ref 1 00000100:00100000:1.0:1789760733.588638:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722870080309 time 722919568482 state 0x0 ref 1 00000100:00100000:1.0:1789760733.588652:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484210048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760733.588716:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20668416, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:1.0:1789760733.588727:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.588749:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:1.0:1789760733.588753:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760733.588761:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20668416->20672511] (req 20668416->20672511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.588776:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20668416->20672511] (req 20668416->20672511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760733.614574:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972364 is committed 00002000:00010000:0.0:1789760733.614768:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20668416 -> 20672512 00010000:00010000:0.0:1789760733.614788:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20668416->20672511] (req 20668416->20672511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.614804:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20668416->20672511] (req 20668416->20672511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.614814:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20668416->20672511] (req 20668416->20672511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.614823:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760733.614829:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20668416->20672511] (req 20668416->20672511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.614840:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20668416->20672511] (req 20668416->20672511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.614870:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20668416->20672511] (req 20668416->20672511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.614879:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962083b7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20668416->20672511] (req 20668416->20672511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760733.615123:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484210048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26466us (26717us total) trans 4294972364 rc 0/0 00000100:00100000:0.0:1789760733.615137:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5075 00000100:00100000:0.0:1789760733.615150:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722870080309 time 722919568482 state 0x2 ref 0 00000100:00100000:1.0:1789760733.617787:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760733.617990:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484210176 00000100:00100000:3.0:1789760733.618004:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484210176 00000100:00100000:3.0:1789760733.618198:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722870080309 time 722919568482 state 0x0 ref 1 00000100:00100000:3.0:1789760733.618224:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722919828898 time 722919568482 state 0x1 ref 1 00000100:00100000:3.0:1789760733.618252:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722919828898 time 722949182862 state 0x0 ref 1 00000100:00100000:3.0:1789760733.618267:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484210176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760733.618333:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20672512, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:3.0:1789760733.618342:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.618494:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=722s v=(0 0) 00010000:00010000:3.0:1789760733.618499:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760733.618506:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20672512->20676607] (req 20672512->20676607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.618520:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20672512->20676607] (req 20672512->20676607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760733.666938:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972365 is committed 00002000:00010000:3.0:1789760733.667222:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20672512 -> 20676608 00010000:00010000:3.0:1789760733.667245:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20672512->20676607] (req 20672512->20676607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.667261:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20672512->20676607] (req 20672512->20676607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.667272:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20672512->20676607] (req 20672512->20676607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.667279:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760733.667286:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20672512->20676607] (req 20672512->20676607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.667297:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20672512->20676607] (req 20672512->20676607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.667321:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20672512->20676607] (req 20672512->20676607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.667329:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083be lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20672512->20676607] (req 20672512->20676607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760733.667503:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484210176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49232us (49723us total) trans 4294972365 rc 0/0 00000100:00100000:3.0:1789760733.667522:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5076 00000100:00100000:3.0:1789760733.667534:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722919828898 time 722949182862 state 0x2 ref 0 00000100:00100000:3.0:1789760733.671642:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760733.671782:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484210304 00000100:00100000:0.0:1789760733.671795:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484210304 00000100:00100000:0.0:1789760733.671875:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722919828898 time 722949182862 state 0x0 ref 1 00000100:00100000:0.0:1789760733.671894:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722949443278 time 722949182862 state 0x1 ref 1 00000100:00100000:0.0:1789760733.671935:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722949443278 time 723002864989 state 0x0 ref 1 00000100:00100000:0.0:1789760733.671950:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484210304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760733.672017:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20676608, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:0.0:1789760733.672028:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.672046:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:0.0:1789760733.672053:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760733.672061:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20676608->20680703] (req 20676608->20680703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760733.672129:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20676608->20680703] (req 20676608->20680703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760733.701806:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20676608 -> 20680704 00010000:00010000:1.0:1789760733.701834:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20676608->20680703] (req 20676608->20680703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.701851:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20676608->20680703] (req 20676608->20680703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.701863:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20676608->20680703] (req 20676608->20680703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.701871:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.701877:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20676608->20680703] (req 20676608->20680703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.701887:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20676608->20680703] (req 20676608->20680703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760733.701904:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972366 is committed 00010000:00010000:1.0:1789760733.701919:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20676608->20680703] (req 20676608->20680703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.701927:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962083c5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20676608->20680703] (req 20676608->20680703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.702142:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484210304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30188us (30507us total) trans 4294972366 rc 0/0 00000100:00100000:1.0:1789760733.702154:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5077 00000100:00100000:1.0:1789760733.702177:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722949443278 time 723002864989 state 0x2 ref 0 00000100:00100000:0.0:1789760733.705890:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760733.705983:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484210432 00000100:00100000:1.0:1789760733.705996:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484210432 00000100:00100000:1.0:1789760733.706106:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 722949443278 time 723002864989 state 0x0 ref 1 00000100:00100000:1.0:1789760733.706124:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723003125405 time 723002864989 state 0x1 ref 1 00000100:00100000:1.0:1789760733.706160:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723003125405 time 723037086498 state 0x0 ref 1 00000100:00100000:1.0:1789760733.706174:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484210432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760733.706240:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20680704, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:1.0:1789760733.706250:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.706269:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:1.0:1789760733.706274:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760733.706282:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20680704->20684799] (req 20680704->20684799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.706297:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20680704->20684799] (req 20680704->20684799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760733.741434:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972367 is committed 00002000:00010000:1.0:1789760733.741826:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20680704 -> 20684800 00010000:00010000:1.0:1789760733.741846:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20680704->20684799] (req 20680704->20684799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.741861:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20680704->20684799] (req 20680704->20684799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.741870:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20680704->20684799] (req 20680704->20684799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.741878:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.741883:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20680704->20684799] (req 20680704->20684799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.741893:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20680704->20684799] (req 20680704->20684799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.741919:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20680704->20684799] (req 20680704->20684799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.741934:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083cc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20680704->20684799] (req 20680704->20684799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.742061:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484210432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35881us (36178us total) trans 4294972367 rc 0/0 00000100:00100000:1.0:1789760733.742121:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5078 00000100:00100000:1.0:1789760733.742135:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723003125405 time 723037086498 state 0x2 ref 0 00000100:00100000:2.0:1789760733.745686:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760733.745842:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484210560 00000100:00100000:1.0:1789760733.745856:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484210560 00000100:00100000:1.0:1789760733.745929:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723003125405 time 723037086498 state 0x0 ref 1 00000100:00100000:1.0:1789760733.745948:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723037346914 time 723037086498 state 0x1 ref 1 00000100:00100000:1.0:1789760733.745987:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723037346914 time 723076917446 state 0x0 ref 1 00000100:00100000:1.0:1789760733.746001:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484210560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760733.746094:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20684800, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:1.0:1789760733.746105:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.746125:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:1.0:1789760733.746131:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760733.746138:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20684800->20688895] (req 20684800->20688895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.746370:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20684800->20688895] (req 20684800->20688895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760733.789733:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972368 is committed 00002000:00010000:2.0:1789760733.789859:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20684800 -> 20688896 00010000:00010000:2.0:1789760733.789880:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20684800->20688895] (req 20684800->20688895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.789897:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20684800->20688895] (req 20684800->20688895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.789907:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20684800->20688895] (req 20684800->20688895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.789915:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760733.789921:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20684800->20688895] (req 20684800->20688895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.789938:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20684800->20688895] (req 20684800->20688895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.789966:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20684800->20688895] (req 20684800->20688895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.789975:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962083d3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20684800->20688895] (req 20684800->20688895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760733.790176:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484210560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44170us (44499us total) trans 4294972368 rc 0/0 00000100:00100000:2.0:1789760733.790189:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5079 00000100:00100000:2.0:1789760733.790202:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723037346914 time 723076917446 state 0x2 ref 0 00000100:00100000:0.0:1789760733.795180:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760733.795275:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484210688 00000100:00100000:3.0:1789760733.795289:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484210688 00000100:00100000:3.0:1789760733.795366:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723037346914 time 723076917446 state 0x0 ref 1 00000100:00100000:3.0:1789760733.795399:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723077177862 time 723076917446 state 0x1 ref 1 00000100:00100000:3.0:1789760733.795436:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723077177862 time 723126366493 state 0x0 ref 1 00000100:00100000:3.0:1789760733.795451:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484210688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760733.795518:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20688896, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:3.0:1789760733.795529:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.795551:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:3.0:1789760733.795556:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760733.795565:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20688896->20692991] (req 20688896->20692991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.795581:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20688896->20692991] (req 20688896->20692991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760733.826125:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972369 is committed 00002000:00010000:3.0:1789760733.826356:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20688896 -> 20692992 00010000:00010000:3.0:1789760733.826386:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20688896->20692991] (req 20688896->20692991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.826402:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20688896->20692991] (req 20688896->20692991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.826412:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20688896->20692991] (req 20688896->20692991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.826420:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760733.826427:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20688896->20692991] (req 20688896->20692991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.826438:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20688896->20692991] (req 20688896->20692991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.826530:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20688896->20692991] (req 20688896->20692991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.826539:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083da lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20688896->20692991] (req 20688896->20692991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760733.826706:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484210688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31250us (31534us total) trans 4294972369 rc 0/0 00000100:00100000:3.0:1789760733.826724:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5080 00000100:00100000:3.0:1789760733.826737:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723077177862 time 723126366493 state 0x2 ref 0 00000100:00100000:0.0:1789760733.830736:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760733.831471:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484210816 00000100:00100000:1.0:1789760733.831487:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484210816 00000100:00100000:1.0:1789760733.831563:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723077177862 time 723126366493 state 0x0 ref 1 00000100:00100000:1.0:1789760733.831583:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723126626909 time 723126366493 state 0x1 ref 1 00000100:00100000:1.0:1789760733.831659:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723126626909 time 723162589228 state 0x0 ref 1 00000100:00100000:1.0:1789760733.831675:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484210816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760733.831742:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20692992, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:1.0:1789760733.831753:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.831838:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:1.0:1789760733.831844:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760733.831853:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20692992->20697087] (req 20692992->20697087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.831871:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20692992->20697087] (req 20692992->20697087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760733.869800:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20692992 -> 20697088 00010000:00010000:1.0:1789760733.869829:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20692992->20697087] (req 20692992->20697087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.869849:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20692992->20697087] (req 20692992->20697087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.869860:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20692992->20697087] (req 20692992->20697087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.869869:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.869875:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20692992->20697087] (req 20692992->20697087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.869884:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20692992->20697087] (req 20692992->20697087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.869908:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20692992->20697087] (req 20692992->20697087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.869916:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962083e1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20692992->20697087] (req 20692992->20697087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.870040:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484210816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38360us (39311us total) trans 4294972370 rc 0/0 00000100:00100000:1.0:1789760733.870056:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5081 00000100:00100000:1.0:1789760733.870104:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723126626909 time 723162589228 state 0x2 ref 0 00000001:00080000:2.0:1789760733.870247:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972370 is committed 00000100:00100000:2.0:1789760733.873621:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760733.873716:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484210944 00000100:00100000:1.0:1789760733.873729:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484210944 00000100:00100000:1.0:1789760733.873800:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723126626909 time 723162589228 state 0x0 ref 1 00000100:00100000:1.0:1789760733.873819:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723162849644 time 723162589228 state 0x1 ref 1 00000100:00100000:1.0:1789760733.873898:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723162849644 time 723204827823 state 0x0 ref 1 00000100:00100000:1.0:1789760733.873913:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484210944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760733.873976:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20697088, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:1.0:1789760733.873985:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.874000:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:1.0:1789760733.874004:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760733.874009:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20697088->20701183] (req 20697088->20701183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.874025:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20697088->20701183] (req 20697088->20701183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760733.901902:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972371 is committed 00002000:00010000:2.0:1789760733.902027:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20697088 -> 20701184 00010000:00010000:2.0:1789760733.902046:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20697088->20701183] (req 20697088->20701183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.902061:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20697088->20701183] (req 20697088->20701183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.902101:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20697088->20701183] (req 20697088->20701183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.902115:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760733.902121:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20697088->20701183] (req 20697088->20701183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.902131:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20697088->20701183] (req 20697088->20701183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.902212:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20697088->20701183] (req 20697088->20701183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.902221:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962083e8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20697088->20701183] (req 20697088->20701183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760733.902338:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484210944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28421us (28725us total) trans 4294972371 rc 0/0 00000100:00100000:2.0:1789760733.902349:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5082 00000100:00100000:2.0:1789760733.902360:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723162849644 time 723204827823 state 0x2 ref 0 00000100:00100000:0.0:1789760733.905700:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760733.905782:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484211072 00000100:00100000:2.0:1789760733.905796:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484211072 00000100:00100000:2.0:1789760733.905865:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723162849644 time 723204827823 state 0x0 ref 1 00000100:00100000:2.0:1789760733.905883:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723205088239 time 723204827823 state 0x1 ref 1 00000100:00100000:2.0:1789760733.905921:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723205088239 time 723236851957 state 0x0 ref 1 00000100:00100000:2.0:1789760733.905938:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484211072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760733.906000:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20701184, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:2.0:1789760733.906011:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.906031:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:2.0:1789760733.906037:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760733.906044:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20701184->20705279] (req 20701184->20705279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.906059:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20701184->20705279] (req 20701184->20705279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760733.951056:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20701184 -> 20705280 00010000:00010000:2.0:1789760733.951162:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20701184->20705279] (req 20701184->20705279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.951179:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20701184->20705279] (req 20701184->20705279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.951189:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20701184->20705279] (req 20701184->20705279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.951196:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760733.951201:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20701184->20705279] (req 20701184->20705279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.951211:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20701184->20705279] (req 20701184->20705279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.951235:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20701184->20705279] (req 20701184->20705279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760733.951243:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962083ef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20701184->20705279] (req 20701184->20705279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760733.951384:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484211072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45432us (45682us total) trans 4294972372 rc 0/0 00000100:00100000:2.0:1789760733.951398:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5083 00000100:00100000:2.0:1789760733.951412:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723205088239 time 723236851957 state 0x2 ref 0 00000001:00080000:3.0:1789760733.951433:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972372 is committed 00000100:00100000:0.0:1789760733.954046:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760733.954214:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484211200 00000100:00100000:3.0:1789760733.954229:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484211200 00000100:00100000:3.0:1789760733.954310:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723205088239 time 723236851957 state 0x0 ref 1 00000100:00100000:3.0:1789760733.954329:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723237112373 time 723236851957 state 0x1 ref 1 00000100:00100000:3.0:1789760733.954362:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723237112373 time 723285292590 state 0x0 ref 1 00000100:00100000:3.0:1789760733.954392:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760733.954459:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20705280, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:3.0:1789760733.954469:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.954489:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:3.0:1789760733.954494:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760733.954502:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20705280->20709375] (req 20705280->20709375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760733.954520:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20705280->20709375] (req 20705280->20709375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760733.997212:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972373 is committed 00002000:00010000:1.0:1789760733.997428:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20705280 -> 20709376 00010000:00010000:1.0:1789760733.997450:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20705280->20709375] (req 20705280->20709375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.997538:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20705280->20709375] (req 20705280->20709375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.997549:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20705280->20709375] (req 20705280->20709375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.997556:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760733.997562:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20705280->20709375] (req 20705280->20709375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.997572:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20705280->20709375] (req 20705280->20709375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.997597:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20705280->20709375] (req 20705280->20709375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760733.997605:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962083f6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20705280->20709375] (req 20705280->20709375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760733.997726:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43333us (43686us total) trans 4294972373 rc 0/0 00000100:00100000:1.0:1789760733.997738:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5084 00000100:00100000:1.0:1789760733.997751:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723237112373 time 723285292590 state 0x2 ref 0 00000100:00100000:2.0:1789760734.001087:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760734.001152:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484211328 00000100:00100000:1.0:1789760734.001176:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484211328 00000100:00100000:1.0:1789760734.001270:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723237112373 time 723285292590 state 0x0 ref 1 00000100:00100000:1.0:1789760734.001288:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723285553006 time 723285292590 state 0x1 ref 1 00000100:00100000:1.0:1789760734.001322:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723285553006 time 723332252987 state 0x0 ref 1 00000100:00100000:1.0:1789760734.001338:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760734.001413:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20709376, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:1.0:1789760734.001424:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.001443:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:1.0:1789760734.001447:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760734.001455:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20709376->20713471] (req 20709376->20713471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.001596:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20709376->20713471] (req 20709376->20713471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760734.002014:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00010000:00010000:3.0:1789760734.093096:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:1.0:1789760734.095034:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20709376 -> 20713472 00010000:00010000:1.0:1789760734.095061:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20709376->20713471] (req 20709376->20713471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.095095:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20709376->20713471] (req 20709376->20713471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.095121:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20709376->20713471] (req 20709376->20713471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.095129:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760734.095135:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20709376->20713471] (req 20709376->20713471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.095146:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20709376->20713471] (req 20709376->20713471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.095178:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20709376->20713471] (req 20709376->20713471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.095187:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962083fd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20709376->20713471] (req 20709376->20713471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760734.095309:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484211328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 93968us (94249us total) trans 4294972374 rc 0/0 00000100:00100000:1.0:1789760734.095322:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5085 00000100:00100000:1.0:1789760734.095335:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723285553006 time 723332252987 state 0x2 ref 0 00000001:00080000:0.0:1789760734.098050:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972374 is committed 00000100:00100000:0.0:1789760734.102430:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760734.102800:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484211456 00000100:00100000:2.0:1789760734.102814:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484211456 00000100:00100000:2.0:1789760734.103334:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723285553006 time 723332252987 state 0x0 ref 1 00000100:00100000:2.0:1789760734.103354:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723332513403 time 723332252987 state 0x1 ref 1 00000100:00100000:2.0:1789760734.103396:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723332513403 time 723434326551 state 0x0 ref 1 00000100:00100000:2.0:1789760734.103412:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760734.103626:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20713472, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:2.0:1789760734.103638:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.103658:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:2.0:1789760734.103663:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760734.103671:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20713472->20717567] (req 20713472->20717567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.103689:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20713472->20717567] (req 20713472->20717567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760734.191922:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972375 is committed 00002000:00010000:2.0:1789760734.192096:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20713472 -> 20717568 00010000:00010000:2.0:1789760734.192118:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20713472->20717567] (req 20713472->20717567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.192132:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20713472->20717567] (req 20713472->20717567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.192141:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20713472->20717567] (req 20713472->20717567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.192149:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760734.192162:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20713472->20717567] (req 20713472->20717567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.192173:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20713472->20717567] (req 20713472->20717567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.192202:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20713472->20717567] (req 20713472->20717567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.192210:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208404 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20713472->20717567] (req 20713472->20717567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760734.192320:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 88906us (89898us total) trans 4294972375 rc 0/0 00000100:00100000:2.0:1789760734.192332:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5086 00000100:00100000:2.0:1789760734.192344:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723332513403 time 723434326551 state 0x2 ref 0 00000100:00100000:3.0:1789760734.197999:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760734.198489:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484211584 00000100:00100000:2.0:1789760734.198503:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484211584 00000100:00100000:2.0:1789760734.198574:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723332513403 time 723434326551 state 0x0 ref 1 00000100:00100000:2.0:1789760734.198592:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723434586967 time 723434326551 state 0x1 ref 1 00000100:00100000:2.0:1789760734.198629:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723434586967 time 723529559372 state 0x0 ref 1 00000100:00100000:2.0:1789760734.198643:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484211584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760734.198712:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20717568, mtime: 0x6aad94dd, atime: 0x0, ctime: 0x6aad94dd, blocks: 0xa007 00010000:00010000:2.0:1789760734.198724:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.198744:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:2.0:1789760734.198935:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760734.198944:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20717568->20721663] (req 20717568->20721663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.198963:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20717568->20721663] (req 20717568->20721663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760734.249783:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972376 is committed 00002000:00010000:3.0:1789760734.250100:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20717568 -> 20721664 00002000:00010000:3.0:1789760734.250106:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760733 -> 1789760734 00002000:00010000:3.0:1789760734.250109:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760733 -> 1789760734 00010000:00010000:3.0:1789760734.250127:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20717568->20721663] (req 20717568->20721663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.250143:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20717568->20721663] (req 20717568->20721663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.250658:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20717568->20721663] (req 20717568->20721663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.250674:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760734.250680:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20717568->20721663] (req 20717568->20721663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.250692:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20717568->20721663] (req 20717568->20721663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.250718:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20717568->20721663] (req 20717568->20721663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.250733:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620840b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20717568->20721663] (req 20717568->20721663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760734.250860:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484211584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52212us (52870us total) trans 4294972376 rc 0/0 00000100:00100000:3.0:1789760734.250872:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5087 00000100:00100000:3.0:1789760734.250883:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723434586967 time 723529559372 state 0x2 ref 0 00000100:00100000:3.0:1789760734.254382:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760734.254827:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484211712 00000100:00100000:1.0:1789760734.254842:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484211712 00000100:00100000:1.0:1789760734.254921:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723434586967 time 723529559372 state 0x0 ref 1 00000100:00100000:1.0:1789760734.254939:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723529819788 time 723529559372 state 0x1 ref 1 00000100:00100000:1.0:1789760734.254979:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723529819788 time 723585909444 state 0x0 ref 1 00000100:00100000:1.0:1789760734.254995:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760734.255099:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20721664, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:1.0:1789760734.255112:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.255132:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:1.0:1789760734.255137:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760734.255145:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20721664->20725759] (req 20721664->20725759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.255344:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20721664->20725759] (req 20721664->20725759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760734.313804:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972377 is committed 00002000:00010000:2.0:1789760734.315326:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20721664 -> 20725760 00010000:00010000:2.0:1789760734.315348:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20721664->20725759] (req 20721664->20725759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.315365:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20721664->20725759] (req 20721664->20725759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.315384:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20721664->20725759] (req 20721664->20725759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.315392:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760734.315397:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20721664->20725759] (req 20721664->20725759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.315407:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20721664->20725759] (req 20721664->20725759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.315432:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20721664->20725759] (req 20721664->20725759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.315446:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208412 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20721664->20725759] (req 20721664->20725759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760734.315582:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60582us (61214us total) trans 4294972377 rc 0/0 00000100:00100000:2.0:1789760734.315595:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5088 00000100:00100000:2.0:1789760734.315608:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723529819788 time 723585909444 state 0x2 ref 0 00000100:00100000:0.0:1789760734.323881:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760734.324201:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484211840 00000100:00100000:1.0:1789760734.324215:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484211840 00000100:00100000:1.0:1789760734.324292:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723529819788 time 723585909444 state 0x0 ref 1 00000100:00100000:1.0:1789760734.324311:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723586169860 time 723585909444 state 0x1 ref 1 00000100:00100000:1.0:1789760734.324489:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723586169860 time 723655418132 state 0x0 ref 1 00000100:00100000:1.0:1789760734.324504:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760734.324581:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20725760, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:1.0:1789760734.324593:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.324614:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:1.0:1789760734.324620:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760734.324628:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20725760->20729855] (req 20725760->20729855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.324659:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20725760->20729855] (req 20725760->20729855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760734.414323:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972378 is committed 00002000:00010000:2.0:1789760734.414464:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20725760 -> 20729856 00010000:00010000:2.0:1789760734.414485:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20725760->20729855] (req 20725760->20729855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.414501:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20725760->20729855] (req 20725760->20729855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.414512:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20725760->20729855] (req 20725760->20729855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.414522:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760734.414528:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20725760->20729855] (req 20725760->20729855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.414539:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20725760->20729855] (req 20725760->20729855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.414569:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20725760->20729855] (req 20725760->20729855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.414578:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208419 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20725760->20729855] (req 20725760->20729855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760734.414703:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 90195us (90833us total) trans 4294972378 rc 0/0 00000100:00100000:2.0:1789760734.414715:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5089 00000100:00100000:2.0:1789760734.414728:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723586169860 time 723655418132 state 0x2 ref 0 00000100:00100000:3.0:1789760734.417562:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760734.420147:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484211968 00000100:00100000:2.0:1789760734.420167:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484211968 00000100:00100000:2.0:1789760734.420251:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723586169860 time 723655418132 state 0x0 ref 1 00000100:00100000:2.0:1789760734.420270:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723655678548 time 723655418132 state 0x1 ref 1 00000100:00100000:2.0:1789760734.421015:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723655678548 time 723751940376 state 0x0 ref 1 00000100:00100000:2.0:1789760734.421038:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484211968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760734.421139:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20729856, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:2.0:1789760734.421149:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.421168:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:2.0:1789760734.421174:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760734.421181:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20729856->20733951] (req 20729856->20733951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.421199:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20729856->20733951] (req 20729856->20733951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760734.467449:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20729856 -> 20733952 00010000:00010000:0.0:1789760734.467477:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20729856->20733951] (req 20729856->20733951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.467493:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20729856->20733951] (req 20729856->20733951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.467504:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20729856->20733951] (req 20729856->20733951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.467513:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760734.467519:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20729856->20733951] (req 20729856->20733951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.467530:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20729856->20733951] (req 20729856->20733951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.467557:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20729856->20733951] (req 20729856->20733951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.467566:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208420 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20729856->20733951] (req 20729856->20733951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760734.467688:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484211968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46651us (50134us total) trans 4294972379 rc 0/0 00000100:00100000:0.0:1789760734.467701:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5090 00000100:00100000:0.0:1789760734.467714:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723655678548 time 723751940376 state 0x2 ref 0 00000001:00080000:0.0:1789760734.467769:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972379 is committed 00000100:00100000:3.0:1789760734.470892:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760734.472119:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484212096 00000100:00100000:2.0:1789760734.472139:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484212096 00000100:00100000:2.0:1789760734.472233:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723655678548 time 723751940376 state 0x0 ref 1 00000100:00100000:2.0:1789760734.472253:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723752200792 time 723751940376 state 0x1 ref 1 00000100:00100000:2.0:1789760734.472289:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723752200792 time 723803219155 state 0x0 ref 1 00000100:00100000:2.0:1789760734.472304:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484212096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760734.472381:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20733952, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:2.0:1789760734.472392:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.472412:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:2.0:1789760734.472417:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760734.472425:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20733952->20738047] (req 20733952->20738047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.472442:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20733952->20738047] (req 20733952->20738047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760734.521208:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972380 is committed 00002000:00010000:0.0:1789760734.521304:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20733952 -> 20738048 00010000:00010000:0.0:1789760734.521326:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20733952->20738047] (req 20733952->20738047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.521351:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20733952->20738047] (req 20733952->20738047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.521666:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20733952->20738047] (req 20733952->20738047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.521676:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760734.521681:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20733952->20738047] (req 20733952->20738047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.521692:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20733952->20738047] (req 20733952->20738047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.521718:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20733952->20738047] (req 20733952->20738047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.521726:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208427 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20733952->20738047] (req 20733952->20738047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760734.521852:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484212096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49545us (50968us total) trans 4294972380 rc 0/0 00000100:00100000:0.0:1789760734.521865:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5091 00000100:00100000:0.0:1789760734.521879:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723752200792 time 723803219155 state 0x2 ref 0 00000100:00100000:2.0:1789760734.528052:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760734.528264:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484212224 00000100:00100000:1.0:1789760734.528278:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484212224 00000100:00100000:1.0:1789760734.528376:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723752200792 time 723803219155 state 0x0 ref 1 00000100:00100000:1.0:1789760734.528395:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723803479571 time 723803219155 state 0x1 ref 1 00000100:00100000:1.0:1789760734.528428:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723803479571 time 723859359306 state 0x0 ref 1 00000100:00100000:1.0:1789760734.528443:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484212224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760734.528637:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20738048, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:1.0:1789760734.528649:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.528670:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:1.0:1789760734.528676:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760734.528684:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20738048->20742143] (req 20738048->20742143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.528700:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20738048->20742143] (req 20738048->20742143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760734.594963:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972381 is committed 00002000:00010000:2.0:1789760734.595131:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20738048 -> 20742144 00010000:00010000:2.0:1789760734.595152:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20738048->20742143] (req 20738048->20742143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.595167:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20738048->20742143] (req 20738048->20742143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.595177:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20738048->20742143] (req 20738048->20742143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.595185:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760734.595190:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20738048->20742143] (req 20738048->20742143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.595201:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20738048->20742143] (req 20738048->20742143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.595229:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20738048->20742143] (req 20738048->20742143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.595237:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620842e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20738048->20742143] (req 20738048->20742143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760734.595388:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484212224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66940us (67344us total) trans 4294972381 rc 0/0 00000100:00100000:2.0:1789760734.595402:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5092 00000100:00100000:2.0:1789760734.595413:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723803479571 time 723859359306 state 0x2 ref 0 00000100:00100000:3.0:1789760734.606467:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760734.609322:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484212352 00000100:00100000:0.0:1789760734.609342:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484212352 00000100:00100000:0.0:1789760734.609454:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723803479571 time 723859359306 state 0x0 ref 1 00000100:00100000:0.0:1789760734.609474:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723859619722 time 723859359306 state 0x1 ref 1 00000100:00100000:0.0:1789760734.609505:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723859619722 time 723940436127 state 0x0 ref 1 00000100:00100000:0.0:1789760734.609523:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484212352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760734.609591:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20742144, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:0.0:1789760734.609603:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.609624:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=723s v=(0 0) 00010000:00010000:0.0:1789760734.609630:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760734.609639:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20742144->20746239] (req 20742144->20746239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.609657:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20742144->20746239] (req 20742144->20746239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760734.671558:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20742144 -> 20746240 00010000:00010000:0.0:1789760734.671586:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20742144->20746239] (req 20742144->20746239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.671604:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20742144->20746239] (req 20742144->20746239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.671615:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20742144->20746239] (req 20742144->20746239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.671623:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760734.671629:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20742144->20746239] (req 20742144->20746239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.671641:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20742144->20746239] (req 20742144->20746239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.671670:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20742144->20746239] (req 20742144->20746239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.671678:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208435 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20742144->20746239] (req 20742144->20746239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760734.671831:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484212352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62305us (65371us total) trans 4294972382 rc 0/0 00000100:00100000:0.0:1789760734.671845:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5093 00000100:00100000:0.0:1789760734.671857:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723859619722 time 723940436127 state 0x2 ref 0 00000001:00080000:1.0:1789760734.672546:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972382 is committed 00000100:00100000:0.0:1789760734.683659:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760734.683915:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484212480 00000100:00100000:2.0:1789760734.683929:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484212480 00000100:00100000:2.0:1789760734.684005:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723859619722 time 723940436127 state 0x0 ref 1 00000100:00100000:2.0:1789760734.684024:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723940696543 time 723940436127 state 0x1 ref 1 00000100:00100000:2.0:1789760734.685280:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723940696543 time 724014994511 state 0x0 ref 1 00000100:00100000:2.0:1789760734.685312:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484212480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760734.685403:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20746240, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:2.0:1789760734.685414:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.685434:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:2.0:1789760734.685440:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760734.685448:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20746240->20750335] (req 20746240->20750335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.685464:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20746240->20750335] (req 20746240->20750335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760734.737735:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972383 is committed 00002000:00010000:0.0:1789760734.737880:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20746240 -> 20750336 00010000:00010000:0.0:1789760734.737899:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20746240->20750335] (req 20746240->20750335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.737914:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20746240->20750335] (req 20746240->20750335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.737924:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20746240->20750335] (req 20746240->20750335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.737938:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760734.737944:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20746240->20750335] (req 20746240->20750335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.737955:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20746240->20750335] (req 20746240->20750335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.737979:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20746240->20750335] (req 20746240->20750335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.737987:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620843c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20746240->20750335] (req 20746240->20750335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760734.738157:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484212480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52840us (54497us total) trans 4294972383 rc 0/0 00000100:00100000:0.0:1789760734.738170:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5094 00000100:00100000:0.0:1789760734.738183:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723940696543 time 724014994511 state 0x2 ref 0 00000100:00100000:2.0:1789760734.745451:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760734.745647:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484212608 00000100:00100000:1.0:1789760734.745662:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484212608 00000100:00100000:1.0:1789760734.745733:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 723940696543 time 724014994511 state 0x0 ref 1 00000100:00100000:1.0:1789760734.745752:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724015254927 time 724014994511 state 0x1 ref 1 00000100:00100000:1.0:1789760734.745790:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724015254927 time 724076719992 state 0x0 ref 1 00000100:00100000:1.0:1789760734.745804:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484212608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760734.745870:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20750336, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:1.0:1789760734.745880:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.745900:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:1.0:1789760734.745904:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760734.745912:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20750336->20754431] (req 20750336->20754431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.745928:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20750336->20754431] (req 20750336->20754431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760734.795554:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972384 is committed 00002000:00010000:1.0:1789760734.795723:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20750336 -> 20754432 00010000:00010000:1.0:1789760734.795744:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20750336->20754431] (req 20750336->20754431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.795760:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20750336->20754431] (req 20750336->20754431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.795823:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20750336->20754431] (req 20750336->20754431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.795831:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760734.795838:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20750336->20754431] (req 20750336->20754431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.795848:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20750336->20754431] (req 20750336->20754431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.795874:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20750336->20754431] (req 20750336->20754431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.795883:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208443 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20750336->20754431] (req 20750336->20754431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760734.796170:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484212608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50360us (50726us total) trans 4294972384 rc 0/0 00000100:00100000:1.0:1789760734.796186:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5095 00000100:00100000:1.0:1789760734.796200:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724015254927 time 724076719992 state 0x2 ref 0 00000100:00100000:3.0:1789760734.802131:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760734.803144:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484212736 00000100:00100000:1.0:1789760734.803160:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484212736 00000100:00100000:1.0:1789760734.803239:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724015254927 time 724076719992 state 0x0 ref 1 00000100:00100000:1.0:1789760734.803259:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724076980408 time 724076719992 state 0x1 ref 1 00000100:00100000:1.0:1789760734.803339:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724076980408 time 724134268323 state 0x0 ref 1 00000100:00100000:1.0:1789760734.803354:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484212736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760734.803433:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20754432, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:1.0:1789760734.803444:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.803465:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:1.0:1789760734.803470:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760734.803478:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20754432->20758527] (req 20754432->20758527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.803495:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20754432->20758527] (req 20754432->20758527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760734.837358:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20754432 -> 20758528 00000001:00080000:3.0:1789760734.837418:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972385 is committed 00010000:00010000:1.0:1789760734.837582:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20754432->20758527] (req 20754432->20758527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.837600:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20754432->20758527] (req 20754432->20758527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.837611:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20754432->20758527] (req 20754432->20758527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.837620:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760734.837625:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20754432->20758527] (req 20754432->20758527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.837636:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20754432->20758527] (req 20754432->20758527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.837666:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20754432->20758527] (req 20754432->20758527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.837675:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620844a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20754432->20758527] (req 20754432->20758527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760734.837849:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484212736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34491us (35725us total) trans 4294972385 rc 0/0 00000100:00100000:1.0:1789760734.837872:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5096 00000100:00100000:1.0:1789760734.837885:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724076980408 time 724134268323 state 0x2 ref 0 00000100:00100000:0.0:1789760734.843097:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760734.843265:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484212864 00000100:00100000:2.0:1789760734.843284:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484212864 00000100:00100000:2.0:1789760734.843359:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724076980408 time 724134268323 state 0x0 ref 1 00000100:00100000:2.0:1789760734.843391:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724134528739 time 724134268323 state 0x1 ref 1 00000100:00100000:2.0:1789760734.843471:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724134528739 time 724174400857 state 0x0 ref 1 00000100:00100000:2.0:1789760734.843485:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484212864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760734.843555:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20758528, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:2.0:1789760734.843566:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.843587:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:2.0:1789760734.843592:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760734.843600:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20758528->20762623] (req 20758528->20762623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.843616:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20758528->20762623] (req 20758528->20762623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760734.871907:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972386 is committed 00002000:00010000:3.0:1789760734.872294:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20758528 -> 20762624 00010000:00010000:3.0:1789760734.872314:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20758528->20762623] (req 20758528->20762623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.872330:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20758528->20762623] (req 20758528->20762623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.872340:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20758528->20762623] (req 20758528->20762623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.872347:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760734.872354:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20758528->20762623] (req 20758528->20762623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.872374:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20758528->20762623] (req 20758528->20762623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.872400:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20758528->20762623] (req 20758528->20762623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.872416:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208451 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20758528->20762623] (req 20758528->20762623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760734.872692:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484212864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29200us (29633us total) trans 4294972386 rc 0/0 00000100:00100000:3.0:1789760734.872705:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5097 00000100:00100000:3.0:1789760734.872718:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724134528739 time 724174400857 state 0x2 ref 0 00000100:00100000:2.0:1789760734.875548:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760734.875673:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484212992 00000100:00100000:0.0:1789760734.875686:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484212992 00000100:00100000:0.0:1789760734.875782:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724134528739 time 724174400857 state 0x0 ref 1 00000100:00100000:0.0:1789760734.875801:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724174661273 time 724174400857 state 0x1 ref 1 00000100:00100000:0.0:1789760734.875839:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724174661273 time 724206769381 state 0x0 ref 1 00000100:00100000:0.0:1789760734.875855:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706cc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484212992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760734.875918:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20762624, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:0.0:1789760734.875928:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.875948:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:0.0:1789760734.875953:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760734.875962:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20762624->20766719] (req 20762624->20766719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.875977:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20762624->20766719] (req 20762624->20766719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760734.911393:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972387 is committed 00002000:00010000:2.0:1789760734.911562:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20762624 -> 20766720 00010000:00010000:2.0:1789760734.911582:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20762624->20766719] (req 20762624->20766719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.911597:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20762624->20766719] (req 20762624->20766719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.911613:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20762624->20766719] (req 20762624->20766719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.911621:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760734.911627:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20762624->20766719] (req 20762624->20766719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.911636:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20762624->20766719] (req 20762624->20766719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.911660:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20762624->20766719] (req 20762624->20766719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760734.911668:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208458 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20762624->20766719] (req 20762624->20766719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760734.911821:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706cc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484212992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35964us (36283us total) trans 4294972387 rc 0/0 00000100:00100000:2.0:1789760734.911833:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5098 00000100:00100000:2.0:1789760734.911846:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724174661273 time 724206769381 state 0x2 ref 0 00000100:00100000:0.0:1789760734.919458:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760734.919791:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484213120 00000100:00100000:1.0:1789760734.919804:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484213120 00000100:00100000:1.0:1789760734.919878:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724174661273 time 724206769381 state 0x0 ref 1 00000100:00100000:1.0:1789760734.919898:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724207029797 time 724206769381 state 0x1 ref 1 00000100:00100000:1.0:1789760734.919936:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724207029797 time 724250866626 state 0x0 ref 1 00000100:00100000:1.0:1789760734.919964:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484213120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760734.920032:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20766720, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:1.0:1789760734.920044:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.920100:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:1.0:1789760734.920107:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760734.920116:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20766720->20770815] (req 20766720->20770815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.920136:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20766720->20770815] (req 20766720->20770815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760734.951525:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20766720 -> 20770816 00000001:00080000:3.0:1789760734.951525:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972388 is committed 00010000:00010000:1.0:1789760734.951648:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20766720->20770815] (req 20766720->20770815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.951659:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20766720->20770815] (req 20766720->20770815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.951667:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20766720->20770815] (req 20766720->20770815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.951672:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760734.951676:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20766720->20770815] (req 20766720->20770815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.951682:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20766720->20770815] (req 20766720->20770815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.951702:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20766720->20770815] (req 20766720->20770815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760734.951707:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620845f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20766720->20770815] (req 20766720->20770815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760734.951860:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484213120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31894us (32411us total) trans 4294972388 rc 0/0 00000100:00100000:1.0:1789760734.951872:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5099 00000100:00100000:1.0:1789760734.951882:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724207029797 time 724250866626 state 0x2 ref 0 00000100:00100000:0.0:1789760734.955516:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760734.955665:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484213248 00000100:00100000:1.0:1789760734.955725:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484213248 00000100:00100000:1.0:1789760734.955899:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724207029797 time 724250866626 state 0x0 ref 1 00000100:00100000:1.0:1789760734.955919:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724251127042 time 724250866626 state 0x1 ref 1 00000100:00100000:1.0:1789760734.955960:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724251127042 time 724286890317 state 0x0 ref 1 00000100:00100000:1.0:1789760734.955974:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484213248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760734.956386:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20770816, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:0.0:1789760734.956400:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.956419:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:0.0:1789760734.956425:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760734.956433:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20770816->20774911] (req 20770816->20774911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.956450:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20770816->20774911] (req 20770816->20774911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760734.983388:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20770816 -> 20774912 00010000:00010000:0.0:1789760734.983415:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20770816->20774911] (req 20770816->20774911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760734.983457:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972389 is committed 00010000:00010000:0.0:1789760734.983643:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20770816->20774911] (req 20770816->20774911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.983656:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20770816->20774911] (req 20770816->20774911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.983663:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760734.983670:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20770816->20774911] (req 20770816->20774911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.983679:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20770816->20774911] (req 20770816->20774911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.983704:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20770816->20774911] (req 20770816->20774911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760734.983713:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396208466 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20770816->20774911] (req 20770816->20774911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760734.983957:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484213248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27978us (28448us total) trans 4294972389 rc 0/0 00000100:00100000:0.0:1789760734.983974:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5100 00000100:00100000:0.0:1789760734.983986:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724251127042 time 724286890317 state 0x2 ref 0 00000100:00100000:2.0:1789760734.987924:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760734.988145:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484213376 00000100:00100000:3.0:1789760734.988324:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484213376 00000100:00100000:3.0:1789760734.988409:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724251127042 time 724286890317 state 0x0 ref 1 00000100:00100000:3.0:1789760734.988528:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724287150733 time 724286890317 state 0x1 ref 1 00000100:00100000:3.0:1789760734.988569:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724287150733 time 724319498732 state 0x0 ref 1 00000100:00100000:3.0:1789760734.988584:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484213376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760734.988651:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20774912, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:3.0:1789760734.988662:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.988683:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:3.0:1789760734.988688:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760734.988696:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20774912->20779007] (req 20774912->20779007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760734.988712:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20774912->20779007] (req 20774912->20779007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760735.062447:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20774912 -> 20779008 00010000:00010000:3.0:1789760735.062482:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20774912->20779007] (req 20774912->20779007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.062499:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20774912->20779007] (req 20774912->20779007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.062508:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20774912->20779007] (req 20774912->20779007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.062516:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760735.062522:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20774912->20779007] (req 20774912->20779007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.062532:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20774912->20779007] (req 20774912->20779007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.062559:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20774912->20779007] (req 20774912->20779007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.062566:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620846d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20774912->20779007] (req 20774912->20779007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760735.062693:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484213376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 74105us (74777us total) trans 4294972390 rc 0/0 00000100:00100000:3.0:1789760735.062706:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5101 00000100:00100000:3.0:1789760735.062718:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724287150733 time 724319498732 state 0x2 ref 0 00000001:00080000:1.0:1789760735.063861:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972390 is committed 00000100:00100000:3.0:1789760735.065806:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760735.065877:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484213504 00000100:00100000:2.0:1789760735.065891:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484213504 00000100:00100000:2.0:1789760735.065964:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724287150733 time 724319498732 state 0x0 ref 1 00000100:00100000:2.0:1789760735.065986:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724319759148 time 724319498732 state 0x1 ref 1 00000100:00100000:2.0:1789760735.066109:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724319759148 time 724397038970 state 0x0 ref 1 00000100:00100000:2.0:1789760735.066125:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484213504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760735.066204:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20779008, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:2.0:1789760735.066216:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.066237:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:2.0:1789760735.066242:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760735.066250:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20779008->20783103] (req 20779008->20783103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.066267:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20779008->20783103] (req 20779008->20783103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760735.099555:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972391 is committed 00002000:00010000:2.0:1789760735.099682:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20779008 -> 20783104 00010000:00010000:2.0:1789760735.099702:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20779008->20783103] (req 20779008->20783103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.099717:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20779008->20783103] (req 20779008->20783103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.099727:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20779008->20783103] (req 20779008->20783103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.099740:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760735.099745:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20779008->20783103] (req 20779008->20783103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.099756:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20779008->20783103] (req 20779008->20783103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.099781:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20779008->20783103] (req 20779008->20783103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.099789:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208474 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20779008->20783103] (req 20779008->20783103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760735.100106:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484213504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33936us (34262us total) trans 4294972391 rc 0/0 00000100:00100000:2.0:1789760735.100121:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5102 00000100:00100000:2.0:1789760735.100134:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724319759148 time 724397038970 state 0x2 ref 0 00000100:00100000:2.0:1789760735.102698:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760735.102805:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484213632 00000100:00100000:1.0:1789760735.102819:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484213632 00000100:00100000:1.0:1789760735.102891:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724319759148 time 724397038970 state 0x0 ref 1 00000100:00100000:1.0:1789760735.102907:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724397299386 time 724397038970 state 0x1 ref 1 00000100:00100000:1.0:1789760735.102937:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724397299386 time 724433866871 state 0x0 ref 1 00000100:00100000:1.0:1789760735.102951:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484213632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760735.103026:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20783104, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:1.0:1789760735.103037:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.103055:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:1.0:1789760735.103061:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760735.103106:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20783104->20787199] (req 20783104->20787199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.103124:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20783104->20787199] (req 20783104->20787199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.116538:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760735.155119:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972392 is committed 00002000:00010000:3.0:1789760735.155702:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20783104 -> 20787200 00010000:00010000:3.0:1789760735.155723:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20783104->20787199] (req 20783104->20787199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.155739:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20783104->20787199] (req 20783104->20787199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.155750:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20783104->20787199] (req 20783104->20787199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.155758:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760735.155764:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20783104->20787199] (req 20783104->20787199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.155775:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20783104->20787199] (req 20783104->20787199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.155801:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20783104->20787199] (req 20783104->20787199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.155810:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620847b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20783104->20787199] (req 20783104->20787199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760735.155936:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484213632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52980us (53245us total) trans 4294972392 rc 0/0 00000100:00100000:3.0:1789760735.155957:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5103 00000100:00100000:3.0:1789760735.155970:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724397299386 time 724433866871 state 0x2 ref 0 00000100:00100000:0.0:1789760735.160002:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760735.160860:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484213760 00000100:00100000:2.0:1789760735.160877:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484213760 00000100:00100000:2.0:1789760735.161202:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724397299386 time 724433866871 state 0x0 ref 1 00000100:00100000:2.0:1789760735.161225:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724434127287 time 724433866871 state 0x1 ref 1 00000100:00100000:2.0:1789760735.161318:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724434127287 time 724492247902 state 0x0 ref 1 00000100:00100000:2.0:1789760735.161336:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484213760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760735.161423:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20787200, mtime: 0x6aad94de, atime: 0x0, ctime: 0x6aad94de, blocks: 0xa007 00010000:00010000:2.0:1789760735.161434:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.161456:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:2.0:1789760735.161462:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760735.161470:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20787200->20791295] (req 20787200->20791295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.161485:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20787200->20791295] (req 20787200->20791295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760735.235421:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972393 is committed 00002000:00010000:1.0:1789760735.235674:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20787200 -> 20791296 00002000:00010000:1.0:1789760735.235679:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760734 -> 1789760735 00002000:00010000:1.0:1789760735.235683:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760734 -> 1789760735 00010000:00010000:1.0:1789760735.235699:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20787200->20791295] (req 20787200->20791295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.235713:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20787200->20791295] (req 20787200->20791295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.235723:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20787200->20791295] (req 20787200->20791295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.235730:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760735.235736:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20787200->20791295] (req 20787200->20791295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.235747:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20787200->20791295] (req 20787200->20791295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.235773:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20787200->20791295] (req 20787200->20791295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.235781:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208482 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20787200->20791295] (req 20787200->20791295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760735.236249:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484213760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 74909us (76265us total) trans 4294972393 rc 0/0 00000100:00100000:1.0:1789760735.236262:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5104 00000100:00100000:1.0:1789760735.236274:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724434127287 time 724492247902 state 0x2 ref 0 00000100:00100000:0.0:1789760735.246600:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760735.246795:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484213888 00000100:00100000:3.0:1789760735.246809:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484213888 00000100:00100000:3.0:1789760735.246954:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724434127287 time 724492247902 state 0x0 ref 1 00000100:00100000:3.0:1789760735.246976:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724492508318 time 724492247902 state 0x1 ref 1 00000100:00100000:3.0:1789760735.247460:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724492508318 time 724578387616 state 0x0 ref 1 00000100:00100000:3.0:1789760735.247550:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484213888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760735.247621:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20791296, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:3.0:1789760735.247632:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.247653:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:3.0:1789760735.247659:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760735.247667:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20791296->20795391] (req 20791296->20795391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.247688:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20791296->20795391] (req 20791296->20795391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760735.351383:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972394 is committed 00002000:00010000:1.0:1789760735.352340:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20791296 -> 20795392 00010000:00010000:1.0:1789760735.352372:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20791296->20795391] (req 20791296->20795391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.352388:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20791296->20795391] (req 20791296->20795391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.352397:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20791296->20795391] (req 20791296->20795391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.352405:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760735.352411:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20791296->20795391] (req 20791296->20795391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.352422:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20791296->20795391] (req 20791296->20795391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.352923:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20791296->20795391] (req 20791296->20795391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.352941:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208489 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20791296->20795391] (req 20791296->20795391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760735.353275:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484213888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 105722us (106680us total) trans 4294972394 rc 0/0 00000100:00100000:1.0:1789760735.353289:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5105 00000100:00100000:1.0:1789760735.353301:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724492508318 time 724578387616 state 0x2 ref 0 00000100:00100000:3.0:1789760735.356690:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760735.356844:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484214016 00000100:00100000:0.0:1789760735.356857:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484214016 00000100:00100000:0.0:1789760735.356944:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724492508318 time 724578387616 state 0x0 ref 1 00000100:00100000:0.0:1789760735.356963:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724578648032 time 724578387616 state 0x1 ref 1 00000100:00100000:0.0:1789760735.356992:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724578648032 time 724687923124 state 0x0 ref 1 00000100:00100000:0.0:1789760735.357005:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484214016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760735.357108:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20795392, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:0.0:1789760735.357120:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.357139:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:0.0:1789760735.357152:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760735.357160:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20795392->20799487] (req 20795392->20799487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.357178:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20795392->20799487] (req 20795392->20799487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760735.437279:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972395 is committed 00002000:00010000:3.0:1789760735.437564:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20795392 -> 20799488 00010000:00010000:3.0:1789760735.437585:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20795392->20799487] (req 20795392->20799487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.437600:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20795392->20799487] (req 20795392->20799487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.437610:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20795392->20799487] (req 20795392->20799487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.437618:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760735.437623:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20795392->20799487] (req 20795392->20799487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.437639:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20795392->20799487] (req 20795392->20799487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.437666:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20795392->20799487] (req 20795392->20799487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.437674:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208490 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20795392->20799487] (req 20795392->20799487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760735.437835:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484214016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 80825us (81154us total) trans 4294972395 rc 0/0 00000100:00100000:3.0:1789760735.437857:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5106 00000100:00100000:3.0:1789760735.437870:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724578648032 time 724687923124 state 0x2 ref 0 00000100:00100000:0.0:1789760735.445476:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760735.450819:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484214144 00000100:00100000:1.0:1789760735.450843:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484214144 00000100:00100000:1.0:1789760735.450939:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724578648032 time 724687923124 state 0x0 ref 1 00000100:00100000:1.0:1789760735.450960:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724688183540 time 724687923124 state 0x1 ref 1 00000100:00100000:1.0:1789760735.451014:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724688183540 time 724781944717 state 0x0 ref 1 00000100:00100000:1.0:1789760735.451031:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484214144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760735.451141:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20799488, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:1.0:1789760735.451152:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.451175:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:1.0:1789760735.451179:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760735.451187:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20799488->20803583] (req 20799488->20803583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.451204:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20799488->20803583] (req 20799488->20803583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760735.511833:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972396 is committed 00002000:00010000:0.0:1789760735.512015:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20799488 -> 20803584 00010000:00010000:0.0:1789760735.512029:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20799488->20803583] (req 20799488->20803583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.512041:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20799488->20803583] (req 20799488->20803583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.512050:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20799488->20803583] (req 20799488->20803583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.512058:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760735.512063:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20799488->20803583] (req 20799488->20803583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.512288:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20799488->20803583] (req 20799488->20803583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.512309:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20799488->20803583] (req 20799488->20803583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.512317:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208497 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20799488->20803583] (req 20799488->20803583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760735.512597:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484214144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61565us (67129us total) trans 4294972396 rc 0/0 00000100:00100000:0.0:1789760735.512609:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5107 00000100:00100000:0.0:1789760735.512619:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724688183540 time 724781944717 state 0x2 ref 0 00000100:00100000:3.0:1789760735.517168:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760735.517244:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484214272 00000100:00100000:2.0:1789760735.517256:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484214272 00000100:00100000:2.0:1789760735.517309:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724688183540 time 724781944717 state 0x0 ref 1 00000100:00100000:2.0:1789760735.517323:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724782205133 time 724781944717 state 0x1 ref 1 00000100:00100000:2.0:1789760735.517366:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724782205133 time 724848281783 state 0x0 ref 1 00000100:00100000:2.0:1789760735.517380:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484214272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760735.517430:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20803584, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:2.0:1789760735.517439:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.517775:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:2.0:1789760735.517780:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760735.517786:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20803584->20807679] (req 20803584->20807679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.517801:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20803584->20807679] (req 20803584->20807679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760735.552106:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20803584 -> 20807680 00010000:00010000:1.0:1789760735.552124:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20803584->20807679] (req 20803584->20807679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.552137:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20803584->20807679] (req 20803584->20807679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760735.552222:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972397 is committed 00010000:00010000:1.0:1789760735.552272:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20803584->20807679] (req 20803584->20807679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.552281:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760735.552286:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20803584->20807679] (req 20803584->20807679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.552300:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20803584->20807679] (req 20803584->20807679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.552321:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20803584->20807679] (req 20803584->20807679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.552329:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620849e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20803584->20807679] (req 20803584->20807679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760735.552630:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484214272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35244us (35465us total) trans 4294972397 rc 0/0 00000100:00100000:1.0:1789760735.552641:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5108 00000100:00100000:1.0:1789760735.552659:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724782205133 time 724848281783 state 0x2 ref 0 00000100:00100000:1.0:1789760735.556602:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760735.556709:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484214400 00000100:00100000:0.0:1789760735.556718:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484214400 00000100:00100000:0.0:1789760735.556776:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724782205133 time 724848281783 state 0x0 ref 1 00000100:00100000:0.0:1789760735.556791:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724848542199 time 724848281783 state 0x1 ref 1 00000100:00100000:0.0:1789760735.556851:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724848542199 time 724887781909 state 0x0 ref 1 00000100:00100000:0.0:1789760735.556865:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484214400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760735.556911:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20807680, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:0.0:1789760735.556920:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.556934:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:0.0:1789760735.556939:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760735.556945:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20807680->20811775] (req 20807680->20811775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760735.556959:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20807680->20811775] (req 20807680->20811775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760735.615824:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20807680 -> 20811776 00010000:00010000:3.0:1789760735.615845:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20807680->20811775] (req 20807680->20811775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.615858:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20807680->20811775] (req 20807680->20811775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.615867:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20807680->20811775] (req 20807680->20811775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.615879:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760735.615884:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20807680->20811775] (req 20807680->20811775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.615893:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20807680->20811775] (req 20807680->20811775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.615911:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20807680->20811775] (req 20807680->20811775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.615919:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962084a5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20807680->20811775] (req 20807680->20811775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760735.616024:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484214400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59158us (59504us total) trans 4294972398 rc 0/0 00000100:00100000:3.0:1789760735.616034:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5109 00000100:00100000:3.0:1789760735.616045:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724848542199 time 724887781909 state 0x2 ref 0 00000001:00080000:1.0:1789760735.617380:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972398 is committed 00000100:00100000:0.0:1789760735.622011:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760735.622122:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484214528 00000100:00100000:3.0:1789760735.622132:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484214528 00000100:00100000:3.0:1789760735.622193:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724848542199 time 724887781909 state 0x0 ref 1 00000100:00100000:3.0:1789760735.622208:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724888042325 time 724887781909 state 0x1 ref 1 00000100:00100000:3.0:1789760735.622231:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724888042325 time 724953162436 state 0x0 ref 1 00000100:00100000:3.0:1789760735.622247:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484214528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760735.622293:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20811776, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:3.0:1789760735.622302:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.622319:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:3.0:1789760735.622324:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760735.622330:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20811776->20815871] (req 20811776->20815871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.622344:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20811776->20815871] (req 20811776->20815871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760735.663846:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972399 is committed 00002000:00010000:3.0:1789760735.663979:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20811776 -> 20815872 00010000:00010000:3.0:1789760735.663999:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20811776->20815871] (req 20811776->20815871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.664013:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20811776->20815871] (req 20811776->20815871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.664024:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20811776->20815871] (req 20811776->20815871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.664033:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760735.664039:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20811776->20815871] (req 20811776->20815871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.664051:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20811776->20815871] (req 20811776->20815871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.664114:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20811776->20815871] (req 20811776->20815871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.664123:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084ac lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20811776->20815871] (req 20811776->20815871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760735.664224:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484214528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41975us (42218us total) trans 4294972399 rc 0/0 00000100:00100000:3.0:1789760735.664235:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5110 00000100:00100000:3.0:1789760735.664248:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724888042325 time 724953162436 state 0x2 ref 0 00000100:00100000:0.0:1789760735.668419:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760735.668649:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484214656 00000100:00100000:1.0:1789760735.668660:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484214656 00000100:00100000:1.0:1789760735.668714:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724888042325 time 724953162436 state 0x0 ref 1 00000100:00100000:1.0:1789760735.668730:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724953422852 time 724953162436 state 0x1 ref 1 00000100:00100000:1.0:1789760735.668760:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724953422852 time 724999691663 state 0x0 ref 1 00000100:00100000:1.0:1789760735.668774:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484214656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760735.668820:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20815872, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:1.0:1789760735.668831:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.668847:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=724s v=(0 0) 00010000:00010000:1.0:1789760735.668852:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760735.668858:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20815872->20819967] (req 20815872->20819967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.668877:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20815872->20819967] (req 20815872->20819967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760735.719481:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972400 is committed 00002000:00010000:1.0:1789760735.719764:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20815872 -> 20819968 00010000:00010000:1.0:1789760735.719778:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20815872->20819967] (req 20815872->20819967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.719792:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20815872->20819967] (req 20815872->20819967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.719801:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20815872->20819967] (req 20815872->20819967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.719808:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760735.719813:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20815872->20819967] (req 20815872->20819967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.719822:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20815872->20819967] (req 20815872->20819967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.719843:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20815872->20819967] (req 20815872->20819967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.719850:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962084b3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20815872->20819967] (req 20815872->20819967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760735.719947:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484214656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51171us (51533us total) trans 4294972400 rc 0/0 00000100:00100000:1.0:1789760735.719957:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5111 00000100:00100000:1.0:1789760735.719968:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724953422852 time 724999691663 state 0x2 ref 0 00000100:00100000:1.0:1789760735.723319:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760735.723412:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484214784 00000100:00100000:2.0:1789760735.723422:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484214784 00000100:00100000:2.0:1789760735.723486:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724953422852 time 724999691663 state 0x0 ref 1 00000100:00100000:2.0:1789760735.723502:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724999952079 time 724999691663 state 0x1 ref 1 00000100:00100000:2.0:1789760735.723531:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724999952079 time 725054462817 state 0x0 ref 1 00000100:00100000:2.0:1789760735.723543:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484214784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760735.723594:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20819968, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:2.0:1789760735.723603:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.723618:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:2.0:1789760735.723623:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760735.723630:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20819968->20824063] (req 20819968->20824063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.723649:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20819968->20824063] (req 20819968->20824063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760735.751100:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972401 is committed 00002000:00010000:1.0:1789760735.751405:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20819968 -> 20824064 00010000:00010000:1.0:1789760735.751421:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20819968->20824063] (req 20819968->20824063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.751434:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20819968->20824063] (req 20819968->20824063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.751512:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20819968->20824063] (req 20819968->20824063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.751522:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760735.751528:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20819968->20824063] (req 20819968->20824063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.751538:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20819968->20824063] (req 20819968->20824063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.751559:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20819968->20824063] (req 20819968->20824063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.751573:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962084ba lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20819968->20824063] (req 20819968->20824063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760735.751703:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484214784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28156us (28387us total) trans 4294972401 rc 0/0 00000100:00100000:1.0:1789760735.751714:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5112 00000100:00100000:1.0:1789760735.751724:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724999952079 time 725054462817 state 0x2 ref 0 00000100:00100000:2.0:1789760735.755607:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760735.755689:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484214912 00000100:00100000:3.0:1789760735.755698:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484214912 00000100:00100000:3.0:1789760735.755751:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 724999952079 time 725054462817 state 0x0 ref 1 00000100:00100000:3.0:1789760735.755767:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725054723233 time 725054462817 state 0x1 ref 1 00000100:00100000:3.0:1789760735.755832:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725054723233 time 725086762488 state 0x0 ref 1 00000100:00100000:3.0:1789760735.755845:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484214912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760735.755895:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20824064, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:3.0:1789760735.755904:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.755920:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760735.755924:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760735.755930:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20824064->20828159] (req 20824064->20828159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.755943:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20824064->20828159] (req 20824064->20828159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760735.789250:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972402 is committed 00002000:00010000:3.0:1789760735.789416:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20824064 -> 20828160 00010000:00010000:3.0:1789760735.789432:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20824064->20828159] (req 20824064->20828159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.789548:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20824064->20828159] (req 20824064->20828159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.789560:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20824064->20828159] (req 20824064->20828159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.789567:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760735.789572:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20824064->20828159] (req 20824064->20828159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.789581:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20824064->20828159] (req 20824064->20828159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.789603:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20824064->20828159] (req 20824064->20828159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.789611:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084c1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20824064->20828159] (req 20824064->20828159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760735.789704:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484214912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33856us (34102us total) trans 4294972402 rc 0/0 00000100:00100000:3.0:1789760735.789715:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5113 00000100:00100000:3.0:1789760735.789726:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725054723233 time 725086762488 state 0x2 ref 0 00000100:00100000:0.0:1789760735.793396:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760735.793555:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484215040 00000100:00100000:3.0:1789760735.793565:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484215040 00000100:00100000:3.0:1789760735.793621:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725054723233 time 725086762488 state 0x0 ref 1 00000100:00100000:3.0:1789760735.793636:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725087022904 time 725086762488 state 0x1 ref 1 00000100:00100000:3.0:1789760735.793708:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725087022904 time 725124638615 state 0x0 ref 1 00000100:00100000:3.0:1789760735.793722:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484215040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760735.793770:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20828160, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:3.0:1789760735.793780:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.793796:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760735.793802:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760735.793808:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20828160->20832255] (req 20828160->20832255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.793822:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20828160->20832255] (req 20828160->20832255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760735.842786:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20828160 -> 20832256 00010000:00010000:3.0:1789760735.842805:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20828160->20832255] (req 20828160->20832255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.842818:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20828160->20832255] (req 20828160->20832255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.842826:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20828160->20832255] (req 20828160->20832255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.842833:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760735.842838:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20828160->20832255] (req 20828160->20832255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.842846:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20828160->20832255] (req 20828160->20832255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.842865:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20828160->20832255] (req 20828160->20832255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.842877:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084c8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20828160->20832255] (req 20828160->20832255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760735.842968:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484215040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49243us (49576us total) trans 4294972403 rc 0/0 00000100:00100000:3.0:1789760735.842979:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5114 00000100:00100000:3.0:1789760735.842990:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725087022904 time 725124638615 state 0x2 ref 0 00000001:00080000:1.0:1789760735.843434:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972403 is committed 00000100:00100000:1.0:1789760735.846935:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760735.849922:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484215168 00000100:00100000:2.0:1789760735.849938:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484215168 00000100:00100000:2.0:1789760735.850004:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725087022904 time 725124638615 state 0x0 ref 1 00000100:00100000:2.0:1789760735.850020:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725124899031 time 725124638615 state 0x1 ref 1 00000100:00100000:2.0:1789760735.850095:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725124899031 time 725181025109 state 0x0 ref 1 00000100:00100000:2.0:1789760735.850108:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484215168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760735.850156:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20832256, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:2.0:1789760735.850164:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.850179:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:2.0:1789760735.850183:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760735.850190:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20832256->20836351] (req 20832256->20836351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.850203:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20832256->20836351] (req 20832256->20836351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760735.925581:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20832256 -> 20836352 00010000:00010000:2.0:1789760735.925601:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20832256->20836351] (req 20832256->20836351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.925615:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20832256->20836351] (req 20832256->20836351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.925625:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20832256->20836351] (req 20832256->20836351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.925631:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760735.925636:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20832256->20836351] (req 20832256->20836351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.925646:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20832256->20836351] (req 20832256->20836351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.925664:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20832256->20836351] (req 20832256->20836351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.925673:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962084cf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20832256->20836351] (req 20832256->20836351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760735.925769:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484215168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 75659us (78837us total) trans 4294972404 rc 0/0 00000100:00100000:2.0:1789760735.925783:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5115 00000100:00100000:2.0:1789760735.925793:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725124899031 time 725181025109 state 0x2 ref 0 00000001:00080000:2.0:1789760735.925928:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972404 is committed 00000100:00100000:2.0:1789760735.934008:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760735.934113:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484215296 00000100:00100000:1.0:1789760735.934123:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484215296 00000100:00100000:1.0:1789760735.934191:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725124899031 time 725181025109 state 0x0 ref 1 00000100:00100000:1.0:1789760735.934207:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725181285525 time 725181025109 state 0x1 ref 1 00000100:00100000:1.0:1789760735.934230:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725181285525 time 725265161584 state 0x0 ref 1 00000100:00100000:1.0:1789760735.934242:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484215296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760735.934299:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20836352, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:1.0:1789760735.934309:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.934325:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:1.0:1789760735.934329:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760735.934335:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20836352->20840447] (req 20836352->20840447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760735.934348:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20836352->20840447] (req 20836352->20840447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760735.964796:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20836352 -> 20840448 00010000:00010000:2.0:1789760735.964822:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20836352->20840447] (req 20836352->20840447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.964836:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20836352->20840447] (req 20836352->20840447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.964846:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20836352->20840447] (req 20836352->20840447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.964853:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760735.964858:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20836352->20840447] (req 20836352->20840447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.964869:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20836352->20840447] (req 20836352->20840447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.964887:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20836352->20840447] (req 20836352->20840447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760735.964895:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf38263962084d6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20836352->20840447] (req 20836352->20840447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760735.965194:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484215296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30948us (31191us total) trans 4294972405 rc 0/0 00000100:00100000:2.0:1789760735.965207:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5116 00000100:00100000:2.0:1789760735.965217:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725181285525 time 725265161584 state 0x2 ref 0 00000001:00080000:2.0:1789760735.965263:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972405 is committed 00000100:00100000:2.0:1789760735.968536:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760735.968808:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484215424 00000100:00100000:3.0:1789760735.968825:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484215424 00000100:00100000:3.0:1789760735.968976:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725181285525 time 725265161584 state 0x0 ref 1 00000100:00100000:3.0:1789760735.968995:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725265422000 time 725265161584 state 0x1 ref 1 00000100:00100000:3.0:1789760735.969029:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725265422000 time 725299959608 state 0x0 ref 1 00000100:00100000:3.0:1789760735.969043:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484215424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760735.969967:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20840448, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:3.0:1789760735.969983:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.970006:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760735.970011:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760735.970018:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20840448->20844543] (req 20840448->20844543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760735.970040:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20840448->20844543] (req 20840448->20844543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760736.021652:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972406 is committed 00002000:00010000:2.0:1789760736.021761:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20840448 -> 20844544 00010000:00010000:2.0:1789760736.021776:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20840448->20844543] (req 20840448->20844543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.021790:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20840448->20844543] (req 20840448->20844543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.021799:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20840448->20844543] (req 20840448->20844543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.021808:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.021813:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20840448->20844543] (req 20840448->20844543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.021822:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20840448->20844543] (req 20840448->20844543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.021841:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20840448->20844543] (req 20840448->20844543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.021850:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084dd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20840448->20844543] (req 20840448->20844543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.021981:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484215424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52935us (53447us total) trans 4294972406 rc 0/0 00000100:00100000:2.0:1789760736.021990:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5117 00000100:00100000:2.0:1789760736.022000:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725265422000 time 725299959608 state 0x2 ref 0 00000100:00100000:1.0:1789760736.025688:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760736.025839:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484215552 00000100:00100000:3.0:1789760736.025849:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484215552 00000100:00100000:3.0:1789760736.025923:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725265422000 time 725299959608 state 0x0 ref 1 00000100:00100000:3.0:1789760736.025939:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725300220024 time 725299959608 state 0x1 ref 1 00000100:00100000:3.0:1789760736.025967:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725300220024 time 725356898064 state 0x0 ref 1 00000100:00100000:3.0:1789760736.025979:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484215552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760736.026025:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20844544, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:3.0:1789760736.026034:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.026049:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760736.026054:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760736.026060:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20844544->20848639] (req 20844544->20848639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.026115:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20844544->20848639] (req 20844544->20848639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1789760736.029868:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760736.061776:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972407 is committed 00002000:00010000:3.0:1789760736.061965:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20844544 -> 20848640 00010000:00010000:3.0:1789760736.061980:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20844544->20848639] (req 20844544->20848639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.061992:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20844544->20848639] (req 20844544->20848639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.062001:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20844544->20848639] (req 20844544->20848639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.062009:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760736.062013:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20844544->20848639] (req 20844544->20848639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.062022:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20844544->20848639] (req 20844544->20848639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.062041:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20844544->20848639] (req 20844544->20848639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.062049:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962084e4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20844544->20848639] (req 20844544->20848639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760736.062305:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484215552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36323us (36622us total) trans 4294972407 rc 0/0 00000100:00100000:3.0:1789760736.062319:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5118 00000100:00100000:3.0:1789760736.062328:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725300220024 time 725356898064 state 0x2 ref 0 00000100:00100000:2.0:1789760736.066544:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760736.067002:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484215680 00000100:00100000:3.0:1789760736.067013:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484215680 00000100:00100000:3.0:1789760736.067097:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725300220024 time 725356898064 state 0x0 ref 1 00000100:00100000:3.0:1789760736.067114:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725357158480 time 725356898064 state 0x1 ref 1 00000100:00100000:3.0:1789760736.067148:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725357158480 time 725398072277 state 0x0 ref 1 00000100:00100000:3.0:1789760736.067161:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484215680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760736.067206:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20848640, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:3.0:1789760736.067215:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.067229:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760736.067234:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760736.067240:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20848640->20852735] (req 20848640->20852735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.067254:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20848640->20852735] (req 20848640->20852735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760736.108771:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972408 is committed 00002000:00010000:2.0:1789760736.108834:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20848640 -> 20852736 00010000:00010000:2.0:1789760736.108849:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20848640->20852735] (req 20848640->20852735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.108863:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20848640->20852735] (req 20848640->20852735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.108872:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20848640->20852735] (req 20848640->20852735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.108880:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.108884:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20848640->20852735] (req 20848640->20852735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.108898:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20848640->20852735] (req 20848640->20852735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.108917:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20848640->20852735] (req 20848640->20852735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.108925:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962084eb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20848640->20852735] (req 20848640->20852735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.109025:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484215680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41862us (42487us total) trans 4294972408 rc 0/0 00000100:00100000:2.0:1789760736.109035:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5119 00000100:00100000:2.0:1789760736.109045:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725357158480 time 725398072277 state 0x2 ref 0 00000100:00100000:2.0:1789760736.112632:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760736.112707:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484215808 00000100:00100000:1.0:1789760736.112718:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484215808 00000100:00100000:1.0:1789760736.112772:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725357158480 time 725398072277 state 0x0 ref 1 00000100:00100000:1.0:1789760736.112787:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725398332693 time 725398072277 state 0x1 ref 1 00000100:00100000:1.0:1789760736.112817:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725398332693 time 725443748435 state 0x0 ref 1 00000100:00100000:1.0:1789760736.112830:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484215808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760736.112877:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20852736, mtime: 0x6aad94df, atime: 0x0, ctime: 0x6aad94df, blocks: 0xa007 00010000:00010000:1.0:1789760736.112885:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.112899:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:1.0:1789760736.112904:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760736.112910:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20852736->20856831] (req 20852736->20856831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.112923:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20852736->20856831] (req 20852736->20856831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.140250:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:2.0:1789760736.146441:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20852736 -> 20856832 00002000:00010000:2.0:1789760736.146451:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760735 -> 1789760736 00002000:00010000:2.0:1789760736.146453:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760735 -> 1789760736 00010000:00010000:2.0:1789760736.146468:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20852736->20856831] (req 20852736->20856831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.146481:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20852736->20856831] (req 20852736->20856831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.146494:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20852736->20856831] (req 20852736->20856831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.146501:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.146506:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20852736->20856831] (req 20852736->20856831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.146515:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20852736->20856831] (req 20852736->20856831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.146534:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20852736->20856831] (req 20852736->20856831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.146540:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962084f2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20852736->20856831] (req 20852736->20856831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.146634:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484215808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33800us (34005us total) trans 4294972409 rc 0/0 00000100:00100000:2.0:1789760736.146645:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5120 00000100:00100000:2.0:1789760736.146654:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725398332693 time 725443748435 state 0x2 ref 0 00000001:00080000:2.0:1789760736.146706:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972409 is committed 00000100:00100000:3.0:1789760736.149738:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760736.149870:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484215936 00000100:00100000:2.0:1789760736.149881:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484215936 00000100:00100000:2.0:1789760736.149936:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725398332693 time 725443748435 state 0x0 ref 1 00000100:00100000:2.0:1789760736.149956:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725444008851 time 725443748435 state 0x1 ref 1 00000100:00100000:2.0:1789760736.149988:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725444008851 time 725480919653 state 0x0 ref 1 00000100:00100000:2.0:1789760736.150002:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484215936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760736.150050:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20856832, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:2.0:1789760736.150058:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.150108:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:2.0:1789760736.150112:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760736.150119:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20856832->20860927] (req 20856832->20860927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.150132:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20856832->20860927] (req 20856832->20860927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760736.192629:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20856832 -> 20860928 00010000:00010000:3.0:1789760736.192649:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20856832->20860927] (req 20856832->20860927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.192662:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20856832->20860927] (req 20856832->20860927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.192672:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20856832->20860927] (req 20856832->20860927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.192680:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760736.192685:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20856832->20860927] (req 20856832->20860927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.192695:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20856832->20860927] (req 20856832->20860927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.192714:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20856832->20860927] (req 20856832->20860927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.192722:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962084f9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20856832->20860927] (req 20856832->20860927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760736.192847:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484215936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42842us (43113us total) trans 4294972410 rc 0/0 00000100:00100000:3.0:1789760736.192857:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5121 00000100:00100000:3.0:1789760736.192867:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725444008851 time 725480919653 state 0x2 ref 0 00000001:00080000:2.0:1789760736.192934:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972410 is committed 00000100:00100000:0.0:1789760736.196367:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760736.196778:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484216064 00000100:00100000:3.0:1789760736.196788:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484216064 00000100:00100000:3.0:1789760736.196841:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725444008851 time 725480919653 state 0x0 ref 1 00000100:00100000:3.0:1789760736.196855:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725481180069 time 725480919653 state 0x1 ref 1 00000100:00100000:3.0:1789760736.196929:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725481180069 time 725527859433 state 0x0 ref 1 00000100:00100000:3.0:1789760736.196942:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484216064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760736.196988:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20860928, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:3.0:1789760736.196997:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.197012:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760736.197016:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760736.197021:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20860928->20865023] (req 20860928->20865023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.197034:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20860928->20865023] (req 20860928->20865023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760736.225960:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972411 is committed 00002000:00010000:2.0:1789760736.226083:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20860928 -> 20865024 00010000:00010000:2.0:1789760736.226099:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20860928->20865023] (req 20860928->20865023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.226111:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20860928->20865023] (req 20860928->20865023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.226120:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20860928->20865023] (req 20860928->20865023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.226128:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.226132:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20860928->20865023] (req 20860928->20865023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.226151:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20860928->20865023] (req 20860928->20865023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.226171:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20860928->20865023] (req 20860928->20865023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.226178:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208500 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20860928->20865023] (req 20860928->20865023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.226272:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484216064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29329us (29919us total) trans 4294972411 rc 0/0 00000100:00100000:2.0:1789760736.226282:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5122 00000100:00100000:2.0:1789760736.226291:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725481180069 time 725527859433 state 0x2 ref 0 00000100:00100000:2.0:1789760736.229709:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760736.230204:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484216192 00000100:00100000:3.0:1789760736.230213:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484216192 00000100:00100000:3.0:1789760736.230267:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725481180069 time 725527859433 state 0x0 ref 1 00000100:00100000:3.0:1789760736.230282:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725528119849 time 725527859433 state 0x1 ref 1 00000100:00100000:3.0:1789760736.230318:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725528119849 time 725561249367 state 0x0 ref 1 00000100:00100000:3.0:1789760736.230331:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484216192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760736.230387:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20865024, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:3.0:1789760736.230396:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.230409:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760736.230413:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760736.230419:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20865024->20869119] (req 20865024->20869119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.230432:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20865024->20869119] (req 20865024->20869119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760736.258517:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972412 is committed 00002000:00010000:3.0:1789760736.258627:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20865024 -> 20869120 00010000:00010000:3.0:1789760736.258643:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20865024->20869119] (req 20865024->20869119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.258658:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20865024->20869119] (req 20865024->20869119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.258668:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20865024->20869119] (req 20865024->20869119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.258675:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760736.258680:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20865024->20869119] (req 20865024->20869119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.258689:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20865024->20869119] (req 20865024->20869119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.258709:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20865024->20869119] (req 20865024->20869119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.258718:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208507 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20865024->20869119] (req 20865024->20869119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760736.258844:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484216192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28509us (29138us total) trans 4294972412 rc 0/0 00000100:00100000:3.0:1789760736.258854:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5123 00000100:00100000:3.0:1789760736.258864:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725528119849 time 725561249367 state 0x2 ref 0 00000100:00100000:3.0:1789760736.261992:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760736.262102:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484216320 00000100:00100000:2.0:1789760736.262113:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484216320 00000100:00100000:2.0:1789760736.262272:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725528119849 time 725561249367 state 0x0 ref 1 00000100:00100000:2.0:1789760736.262288:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725561509783 time 725561249367 state 0x1 ref 1 00000100:00100000:2.0:1789760736.262320:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725561509783 time 725593250691 state 0x0 ref 1 00000100:00100000:2.0:1789760736.262334:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484216320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760736.262392:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20869120, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:2.0:1789760736.262401:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.262416:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:2.0:1789760736.262421:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760736.262428:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20869120->20873215] (req 20869120->20873215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.262542:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20869120->20873215] (req 20869120->20873215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760736.291337:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972413 is committed 00002000:00010000:3.0:1789760736.291513:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20869120 -> 20873216 00010000:00010000:3.0:1789760736.291530:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20869120->20873215] (req 20869120->20873215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.291543:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20869120->20873215] (req 20869120->20873215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.291553:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20869120->20873215] (req 20869120->20873215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.291567:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760736.291573:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20869120->20873215] (req 20869120->20873215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.291582:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20869120->20873215] (req 20869120->20873215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.291600:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20869120->20873215] (req 20869120->20873215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.291608:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620850e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20869120->20873215] (req 20869120->20873215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760736.291701:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484216320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29366us (29714us total) trans 4294972413 rc 0/0 00000100:00100000:3.0:1789760736.291712:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5124 00000100:00100000:3.0:1789760736.291722:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725561509783 time 725593250691 state 0x2 ref 0 00000100:00100000:1.0:1789760736.295813:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760736.295916:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484216448 00000100:00100000:2.0:1789760736.295926:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484216448 00000100:00100000:2.0:1789760736.295979:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725561509783 time 725593250691 state 0x0 ref 1 00000100:00100000:2.0:1789760736.295995:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725593511107 time 725593250691 state 0x1 ref 1 00000100:00100000:2.0:1789760736.296055:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725593511107 time 725626985403 state 0x0 ref 1 00000100:00100000:2.0:1789760736.296088:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484216448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760736.296138:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20873216, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:2.0:1789760736.296157:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.296179:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:2.0:1789760736.296183:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760736.296190:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20873216->20877311] (req 20873216->20877311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.296204:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20873216->20877311] (req 20873216->20877311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760736.331613:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972414 is committed 00002000:00010000:2.0:1789760736.331723:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20873216 -> 20877312 00010000:00010000:2.0:1789760736.331739:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20873216->20877311] (req 20873216->20877311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.331751:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20873216->20877311] (req 20873216->20877311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.331760:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20873216->20877311] (req 20873216->20877311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.331768:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.331773:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20873216->20877311] (req 20873216->20877311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.331782:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20873216->20877311] (req 20873216->20877311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.331801:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20873216->20877311] (req 20873216->20877311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.331810:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208515 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20873216->20877311] (req 20873216->20877311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.331899:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484216448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35829us (36091us total) trans 4294972414 rc 0/0 00000100:00100000:2.0:1789760736.331909:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5125 00000100:00100000:2.0:1789760736.331919:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725593511107 time 725626985403 state 0x2 ref 0 00000100:00100000:0.0:1789760736.335317:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760736.335422:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484216576 00000100:00100000:3.0:1789760736.335434:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484216576 00000100:00100000:3.0:1789760736.335551:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725593511107 time 725626985403 state 0x0 ref 1 00000100:00100000:3.0:1789760736.335567:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725627245819 time 725626985403 state 0x1 ref 1 00000100:00100000:3.0:1789760736.335628:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725627245819 time 725666559172 state 0x0 ref 1 00000100:00100000:3.0:1789760736.335645:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484216576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760736.335692:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20877312, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:3.0:1789760736.335701:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.335717:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760736.335722:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760736.335729:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20877312->20881407] (req 20877312->20881407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.335741:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20877312->20881407] (req 20877312->20881407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760736.364267:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972415 is committed 00002000:00010000:0.0:1789760736.364381:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20877312 -> 20881408 00010000:00010000:0.0:1789760736.364397:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20877312->20881407] (req 20877312->20881407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.364409:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20877312->20881407] (req 20877312->20881407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.364418:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20877312->20881407] (req 20877312->20881407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.364426:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760736.364430:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20877312->20881407] (req 20877312->20881407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.364617:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20877312->20881407] (req 20877312->20881407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.364640:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20877312->20881407] (req 20877312->20881407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.364648:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620851c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20877312->20881407] (req 20877312->20881407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760736.364738:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484216576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29091us (29426us total) trans 4294972415 rc 0/0 00000100:00100000:0.0:1789760736.364748:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5126 00000100:00100000:0.0:1789760736.364757:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725627245819 time 725666559172 state 0x2 ref 0 00000100:00100000:1.0:1789760736.368325:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760736.368417:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484216704 00000100:00100000:3.0:1789760736.368427:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484216704 00000100:00100000:3.0:1789760736.368539:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725627245819 time 725666559172 state 0x0 ref 1 00000100:00100000:3.0:1789760736.368554:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725666819588 time 725666559172 state 0x1 ref 1 00000100:00100000:3.0:1789760736.368587:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725666819588 time 725699517502 state 0x0 ref 1 00000100:00100000:3.0:1789760736.368600:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484216704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760736.368649:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20881408, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:3.0:1789760736.368658:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.368679:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760736.368684:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760736.368690:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20881408->20885503] (req 20881408->20885503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.368705:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20881408->20885503] (req 20881408->20885503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760736.399779:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20881408 -> 20885504 00010000:00010000:2.0:1789760736.399799:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20881408->20885503] (req 20881408->20885503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.399813:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20881408->20885503] (req 20881408->20885503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.399823:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20881408->20885503] (req 20881408->20885503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.399831:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.399835:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20881408->20885503] (req 20881408->20885503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.399844:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20881408->20885503] (req 20881408->20885503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.399869:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20881408->20885503] (req 20881408->20885503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.399877:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208523 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20881408->20885503] (req 20881408->20885503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.399970:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484216704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31368us (31651us total) trans 4294972416 rc 0/0 00000100:00100000:2.0:1789760736.399980:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5127 00000100:00100000:2.0:1789760736.399989:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725666819588 time 725699517502 state 0x2 ref 0 00000001:00080000:1.0:1789760736.400427:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972416 is committed 00000100:00100000:1.0:1789760736.403452:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760736.403497:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484216832 00000100:00100000:2.0:1789760736.403507:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484216832 00000100:00100000:2.0:1789760736.403560:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725666819588 time 725699517502 state 0x0 ref 1 00000100:00100000:2.0:1789760736.403575:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725699777918 time 725699517502 state 0x1 ref 1 00000100:00100000:2.0:1789760736.403605:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725699777918 time 725734535802 state 0x0 ref 1 00000100:00100000:2.0:1789760736.403617:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484216832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760736.403666:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20885504, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:2.0:1789760736.403675:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.403690:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:2.0:1789760736.403696:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760736.403702:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20885504->20889599] (req 20885504->20889599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.403715:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20885504->20889599] (req 20885504->20889599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760736.436628:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972417 is committed 00002000:00010000:2.0:1789760736.436726:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20885504 -> 20889600 00010000:00010000:2.0:1789760736.436742:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20885504->20889599] (req 20885504->20889599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.436756:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20885504->20889599] (req 20885504->20889599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.436764:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20885504->20889599] (req 20885504->20889599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.436771:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.436776:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20885504->20889599] (req 20885504->20889599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.436785:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20885504->20889599] (req 20885504->20889599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.436806:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20885504->20889599] (req 20885504->20889599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.436815:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620852a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20885504->20889599] (req 20885504->20889599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.436909:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484216832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33289us (33468us total) trans 4294972417 rc 0/0 00000100:00100000:2.0:1789760736.436925:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5128 00000100:00100000:2.0:1789760736.436935:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725699777918 time 725734535802 state 0x2 ref 0 00000100:00100000:0.0:1789760736.443209:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760736.443311:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484216960 00000100:00100000:2.0:1789760736.443323:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484216960 00000100:00100000:2.0:1789760736.443389:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725699777918 time 725734535802 state 0x0 ref 1 00000100:00100000:2.0:1789760736.443406:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725734796218 time 725734535802 state 0x1 ref 1 00000100:00100000:2.0:1789760736.443499:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725734796218 time 725774427806 state 0x0 ref 1 00000100:00100000:2.0:1789760736.443513:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484216960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760736.443565:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20889600, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:2.0:1789760736.443574:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.443591:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:2.0:1789760736.443595:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760736.443601:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20889600->20893695] (req 20889600->20893695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.443616:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20889600->20893695] (req 20889600->20893695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760736.485025:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972418 is committed 00002000:00010000:2.0:1789760736.485228:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20889600 -> 20893696 00010000:00010000:2.0:1789760736.485243:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20889600->20893695] (req 20889600->20893695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.485256:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20889600->20893695] (req 20889600->20893695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.485266:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20889600->20893695] (req 20889600->20893695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.485273:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.485335:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20889600->20893695] (req 20889600->20893695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.485346:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20889600->20893695] (req 20889600->20893695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.485379:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20889600->20893695] (req 20889600->20893695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.485386:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208531 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20889600->20893695] (req 20889600->20893695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.485579:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484216960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42063us (42374us total) trans 4294972418 rc 0/0 00000100:00100000:2.0:1789760736.485592:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5129 00000100:00100000:2.0:1789760736.485601:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725734796218 time 725774427806 state 0x2 ref 0 00000100:00100000:1.0:1789760736.495561:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760736.495631:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484217088 00000100:00100000:0.0:1789760736.495643:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484217088 00000100:00100000:0.0:1789760736.495699:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725734796218 time 725774427806 state 0x0 ref 1 00000100:00100000:0.0:1789760736.495714:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725774688222 time 725774427806 state 0x1 ref 1 00000100:00100000:0.0:1789760736.495745:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725774688222 time 725826675801 state 0x0 ref 1 00000100:00100000:0.0:1789760736.495759:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484217088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760736.495807:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20893696, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:0.0:1789760736.495816:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.495841:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:0.0:1789760736.495845:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760736.495851:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20893696->20897791] (req 20893696->20897791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.495870:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20893696->20897791] (req 20893696->20897791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760736.559721:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972419 is committed 00002000:00010000:2.0:1789760736.561404:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20893696 -> 20897792 00010000:00010000:2.0:1789760736.561423:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20893696->20897791] (req 20893696->20897791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.561435:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20893696->20897791] (req 20893696->20897791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.561454:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20893696->20897791] (req 20893696->20897791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.561461:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.561466:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20893696->20897791] (req 20893696->20897791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.561474:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20893696->20897791] (req 20893696->20897791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.561494:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20893696->20897791] (req 20893696->20897791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.561502:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208538 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20893696->20897791] (req 20893696->20897791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.561621:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484217088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65861us (66065us total) trans 4294972419 rc 0/0 00000100:00100000:2.0:1789760736.561631:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5130 00000100:00100000:2.0:1789760736.561640:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725774688222 time 725826675801 state 0x2 ref 0 00000100:00100000:1.0:1789760736.564920:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760736.565010:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484217216 00000100:00100000:3.0:1789760736.565019:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484217216 00000100:00100000:3.0:1789760736.565109:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725774688222 time 725826675801 state 0x0 ref 1 00000100:00100000:3.0:1789760736.565123:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725826936217 time 725826675801 state 0x1 ref 1 00000100:00100000:3.0:1789760736.565157:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725826936217 time 725896087899 state 0x0 ref 1 00000100:00100000:3.0:1789760736.565170:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484217216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760736.565216:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20897792, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:3.0:1789760736.565225:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.565239:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:3.0:1789760736.565243:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760736.565249:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20897792->20901887] (req 20897792->20901887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.565266:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20897792->20901887] (req 20897792->20901887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760736.600961:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972420 is committed 00002000:00010000:2.0:1789760736.601055:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20897792 -> 20901888 00010000:00010000:2.0:1789760736.601132:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20897792->20901887] (req 20897792->20901887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.601153:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20897792->20901887] (req 20897792->20901887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.601161:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20897792->20901887] (req 20897792->20901887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.601169:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.601174:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20897792->20901887] (req 20897792->20901887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.601184:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20897792->20901887] (req 20897792->20901887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.601203:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20897792->20901887] (req 20897792->20901887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.601217:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620853f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20897792->20901887] (req 20897792->20901887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.601311:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484217216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36139us (36401us total) trans 4294972420 rc 0/0 00000100:00100000:2.0:1789760736.601322:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5131 00000100:00100000:2.0:1789760736.601331:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725826936217 time 725896087899 state 0x2 ref 0 00000100:00100000:3.0:1789760736.609203:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760736.609318:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484217344 00000100:00100000:2.0:1789760736.609328:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484217344 00000100:00100000:2.0:1789760736.609396:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725826936217 time 725896087899 state 0x0 ref 1 00000100:00100000:2.0:1789760736.609411:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725896348315 time 725896087899 state 0x1 ref 1 00000100:00100000:2.0:1789760736.609445:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725896348315 time 725940376250 state 0x0 ref 1 00000100:00100000:2.0:1789760736.609457:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484217344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760736.609508:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20901888, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:2.0:1789760736.609518:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.609537:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:2.0:1789760736.609543:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760736.609550:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20901888->20905983] (req 20901888->20905983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.609571:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20901888->20905983] (req 20901888->20905983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760736.652479:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9470380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3480:1876699508917760:0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.652521:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9470e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3481:1876699508917888:0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.652547:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760736.652550:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760736.652590:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3481:1876699508918400:0@lo:400:kworker.0 00000100:00100000:0.0:1789760736.652923:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508918016:0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.653276:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e85180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508918272:0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.653300:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760736.653315:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e85c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508918784:0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.653329:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760736.653340:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df82c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508919296:0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.653364:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760736.653383:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1789760736.653389:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508919424:0@lo:13:osp-pre-1-1.0 00000100:00100000:1.0:1789760736.653414:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760736.653490:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508918272 00000100:00100000:3.0:1789760736.653498:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508918272 00000100:00100000:3.0:1789760736.653544:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 435 00000100:00100000:3.0:1789760736.653555:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508918272:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760736.653608:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508918272:12345-0@lo:400:kworker.0 Request processed in 56us (306us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760736.653614:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 435 00000100:00100000:2.0:1789760736.653632:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760736.653642:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508919424 00000100:00100000:3.0:1789760736.653647:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508919424 00000100:00100000:1.0:1789760736.653651:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e87b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508918144:0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.653663:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3481:1876699508918912:0@lo:400:kworker.0 00000100:00100000:3.0:1789760736.653692:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 376 00000100:00100000:1.0:1789760736.653693:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760736.653706:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508919424:12345-0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760736.653719:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508917760 00000100:00100000:2.0:1789760736.653724:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508917760 00000100:00100000:1.0:1789760736.653733:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508918656:0@lo:400:kworker.0 00002000:00000024:3.0:1789760736.653736:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760736.653759:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 476 00000100:00100000:2.0:1789760736.653769:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b6050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3480:x1876699508917760:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760736.653777:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508919424:12345-0@lo:13:osp-pre-1-1.0 Request processed in 75us (361us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760736.653784:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 376 00000100:00100000:3.0:1789760736.653805:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508917888 00000100:00100000:3.0:1789760736.653809:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508917888 00000100:00100000:3.0:1789760736.653833:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508918400 00000100:00100000:3.0:1789760736.653835:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508918400 00000100:00100000:3.0:1789760736.653852:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508918144 00000100:00100000:3.0:1789760736.653855:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508918144 00000100:00100000:3.0:1789760736.653873:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 905 00000100:00100000:3.0:1789760736.653881:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3481:x1876699508917888:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.653898:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760736.653923:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df82e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508919168:0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.653946:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760736.654035:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b6050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3480:x1876699508917760:12345-0@lo:400:kworker.0 Request processed in 267us (1487us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.654043:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 476 00000100:00100000:2.0:1789760736.654097:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 906 00000100:00100000:2.0:1789760736.654107:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3481:x1876699508918400:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.654142:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3481:x1876699508918400:12345-0@lo:400:kworker.0 Request processed in 38us (512us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.654148:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 906 00000100:00100000:2.0:1789760736.654164:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 907 00000100:00100000:2.0:1789760736.654172:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508918144:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.654206:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508918144:12345-0@lo:400:kworker.0 Request processed in 35us (516us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.654213:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 907 00000100:00100000:2.0:1789760736.654232:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508918784 00000100:00100000:2.0:1789760736.654236:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508918784 00000100:00100000:2.0:1789760736.654265:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508919296 00000100:00100000:2.0:1789760736.654268:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508919296 00000100:00100000:2.0:1789760736.654285:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508918656 00000100:00100000:2.0:1789760736.654288:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508918656 00000100:00100000:2.0:1789760736.654303:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508919168 00000100:00100000:2.0:1789760736.654306:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508919168 00000100:00100000:2.0:1789760736.654318:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 316 00000100:00100000:2.0:1789760736.654325:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508918784:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.654375:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508918784:12345-0@lo:400:kworker.0 Request processed in 50us (1044us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.654383:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 316 00000100:00100000:2.0:1789760736.654394:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 317 00000100:00100000:2.0:1789760736.654401:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508919296:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.654497:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508919296:12345-0@lo:400:kworker.0 Request processed in 97us (1132us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.654504:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 317 00000100:00100000:2.0:1789760736.654516:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 318 00000100:00100000:2.0:1789760736.654523:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508918656:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.654548:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508918656:12345-0@lo:400:kworker.0 Request processed in 26us (651us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.654557:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 318 00000100:00100000:2.0:1789760736.654569:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 319 00000100:00100000:2.0:1789760736.654576:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508919168:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.654664:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508919168:12345-0@lo:400:kworker.0 Request processed in 86us (714us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.654672:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 319 00000100:00100000:0.0:1789760736.654694:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760736.654782:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508918016 00000100:00100000:1.0:1789760736.654788:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508918016 00000100:00100000:0.0:1789760736.654797:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508918528:0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.654822:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 436 00000100:00100000:0.0:1789760736.654829:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760736.654833:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457a680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508918016:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760736.654852:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508919040:0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.654880:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457a680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508918016:12345-0@lo:400:kworker.0 Request processed in 50us (186us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760736.654886:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760736.654894:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 436 00000100:00100000:0.0:1789760736.654907:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c13100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508919552:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760736.654933:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760736.654956:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e87b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508918144:0@lo:400:kworker.0 00000100:00100000:0.0:1789760736.654967:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508918016:0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.654976:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508918656:0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.654991:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df82e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508919168:0@lo:400:kworker.0 00000100:00100000:0.0:1789760736.655002:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508918528 00000100:00100000:0.0:1789760736.655007:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508918528 00000100:00100000:2.0:1789760736.655023:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760736.655031:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508919040 00000100:00100000:1.0:1789760736.655033:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508919552 00000100:00100000:0.0:1789760736.655036:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508919040 00000100:00100000:1.0:1789760736.655040:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508919552 00000100:00100000:2.0:1789760736.655056:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3481:1876699508918400:0@lo:400:kworker.0 00000100:00100000:1.0:1789760736.655091:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 377 00000100:00100000:1.0:1789760736.655101:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508919552:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:1.0:1789760736.655114:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760736.655154:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508919552:12345-0@lo:13:osp-pre-1-0.0 Request processed in 55us (219us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.655160:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508918912 00000100:00100000:1.0:1789760736.655161:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 377 00000100:00100000:2.0:1789760736.655166:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508918912 00000100:00100000:2.0:1789760736.655192:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 908 00000100:00100000:2.0:1789760736.655200:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508918528:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760736.655207:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 909 00000100:00100000:0.0:1789760736.655219:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508919040:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.655237:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508918528:12345-0@lo:400:kworker.0 Request processed in 37us (408us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.655244:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 908 00000100:00100000:2.0:1789760736.655261:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 910 00000100:00100000:2.0:1789760736.655268:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0001_UUID+5:3481:x1876699508918912:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760736.655340:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508919040:12345-0@lo:400:kworker.0 Request processed in 124us (456us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760736.655348:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 909 00000100:00100000:2.0:1789760736.655373:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0001_UUID+5:3481:x1876699508918912:12345-0@lo:400:kworker.0 Request processed in 105us (350us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760736.655381:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 910 00000100:00100000:0.0:1789760736.655402:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c13100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508919552:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760736.655409:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3481:1876699508918912:0@lo:400:kworker.0 00000100:00100000:0.0:1789760736.655422:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508918528:0@lo:400:kworker.0 00000100:00100000:0.0:1789760736.655434:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508919040:0@lo:400:kworker.0 00000100:00080000:2.0:1789760736.655476:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760736.655481:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:2.0:1789760736.655483:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:2.0:1789760736.655485:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1789760736.655509:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508919424:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760736.655521:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e85180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508918272:0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.655530:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e85c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508918784:0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.655539:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df82c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508919296:0@lo:400:kworker.0 00000100:00100000:2.0:1789760736.655548:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9470380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3480:1876699508917760:0@lo:400:kworker.0 00000100:00100000:3.0:1789760736.655607:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c12a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3481:x1876699508917888:12345-0@lo:400:kworker.0 Request processed in 1725us (3053us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760736.655615:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 905 00000100:00100000:2.0:1789760736.655690:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9470e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3481:1876699508917888:0@lo:400:kworker.0 00000001:00080000:0.0:1789760736.656861:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972421 is committed 00002000:00010000:0.0:1789760736.656919:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20901888 -> 20905984 00010000:00010000:0.0:1789760736.656932:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20901888->20905983] (req 20901888->20905983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.656944:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20901888->20905983] (req 20901888->20905983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.656953:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20901888->20905983] (req 20901888->20905983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.656960:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760736.656964:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20901888->20905983] (req 20901888->20905983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.656973:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20901888->20905983] (req 20901888->20905983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.656992:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20901888->20905983] (req 20901888->20905983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.657000:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208546 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20901888->20905983] (req 20901888->20905983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760736.657536:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484217344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48077us (48468us total) trans 4294972421 rc 0/0 00000100:00100000:0.0:1789760736.657546:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5132 00000100:00100000:0.0:1789760736.657554:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725896348315 time 725940376250 state 0x2 ref 0 00000100:00100000:1.0:1789760736.661679:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760736.661797:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484217472 00000100:00100000:2.0:1789760736.661807:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484217472 00000100:00100000:2.0:1789760736.661865:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725896348315 time 725940376250 state 0x0 ref 1 00000100:00100000:2.0:1789760736.661880:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725940636666 time 725940376250 state 0x1 ref 1 00000100:00100000:2.0:1789760736.661952:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725940636666 time 725992882450 state 0x0 ref 1 00000100:00100000:2.0:1789760736.661965:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484217472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760736.662015:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20905984, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:2.0:1789760736.662025:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.662041:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=725s v=(0 0) 00010000:00010000:2.0:1789760736.662057:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760736.662348:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20905984->20910079] (req 20905984->20910079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.662381:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20905984->20910079] (req 20905984->20910079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760736.692819:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972422 is committed 00002000:00010000:0.0:1789760736.692919:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20905984 -> 20910080 00010000:00010000:0.0:1789760736.692933:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20905984->20910079] (req 20905984->20910079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.692946:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20905984->20910079] (req 20905984->20910079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.692955:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20905984->20910079] (req 20905984->20910079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.692962:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760736.692967:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20905984->20910079] (req 20905984->20910079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.692977:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20905984->20910079] (req 20905984->20910079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.692999:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20905984->20910079] (req 20905984->20910079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.693011:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620854d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20905984->20910079] (req 20905984->20910079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760736.693173:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484217472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31206us (31498us total) trans 4294972422 rc 0/0 00000100:00100000:0.0:1789760736.693182:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5133 00000100:00100000:0.0:1789760736.693191:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725940636666 time 725992882450 state 0x2 ref 0 00000100:00100000:1.0:1789760736.697114:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760736.697190:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484217600 00000100:00100000:0.0:1789760736.697206:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484217600 00000100:00100000:0.0:1789760736.698641:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725940636666 time 725992882450 state 0x0 ref 1 00000100:00100000:0.0:1789760736.698660:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725993142866 time 725992882450 state 0x1 ref 1 00000100:00100000:1.0:1789760736.699454:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725993142866 time 726030379741 state 0x0 ref 1 00000100:00100000:1.0:1789760736.699479:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484217600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760736.699537:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20910080, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:1.0:1789760736.699546:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.699563:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:1.0:1789760736.699568:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760736.699574:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20910080->20914175] (req 20910080->20914175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.699589:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20910080->20914175] (req 20910080->20914175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760736.736365:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972423 is committed 00002000:00010000:1.0:1789760736.736576:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20910080 -> 20914176 00010000:00010000:1.0:1789760736.736671:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20910080->20914175] (req 20910080->20914175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.736688:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20910080->20914175] (req 20910080->20914175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.736700:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20910080->20914175] (req 20910080->20914175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.736708:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760736.736713:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20910080->20914175] (req 20910080->20914175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.736723:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20910080->20914175] (req 20910080->20914175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.736749:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20910080->20914175] (req 20910080->20914175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.736757:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208554 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20910080->20914175] (req 20910080->20914175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760736.736998:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484217600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37521us (39943us total) trans 4294972423 rc 0/0 00000100:00100000:1.0:1789760736.737011:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5134 00000100:00100000:1.0:1789760736.737024:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725993142866 time 726030379741 state 0x2 ref 0 00000100:00100000:1.0:1789760736.741804:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760736.742030:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484217728 00000100:00100000:3.0:1789760736.742044:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484217728 00000100:00100000:2.0:1789760736.742511:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 725993142866 time 726030379741 state 0x0 ref 1 00000100:00100000:2.0:1789760736.742532:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726030640157 time 726030379741 state 0x1 ref 1 00000100:00100000:2.0:1789760736.742681:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726030640157 time 726073610395 state 0x0 ref 1 00000100:00100000:2.0:1789760736.742698:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484217728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760736.742774:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20914176, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:2.0:1789760736.742785:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.742805:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:2.0:1789760736.742810:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760736.742827:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20914176->20918271] (req 20914176->20918271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.742845:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20914176->20918271] (req 20914176->20918271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760736.788848:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972424 is committed 00002000:00010000:3.0:1789760736.789237:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20914176 -> 20918272 00010000:00010000:3.0:1789760736.789258:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20914176->20918271] (req 20914176->20918271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.789275:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20914176->20918271] (req 20914176->20918271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.789285:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20914176->20918271] (req 20914176->20918271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.789294:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760736.789300:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20914176->20918271] (req 20914176->20918271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.789311:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20914176->20918271] (req 20914176->20918271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.789335:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20914176->20918271] (req 20914176->20918271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.789343:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620855b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20914176->20918271] (req 20914176->20918271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760736.789484:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484217728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46783us (47687us total) trans 4294972424 rc 0/0 00000100:00100000:3.0:1789760736.789497:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5135 00000100:00100000:3.0:1789760736.789510:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726030640157 time 726073610395 state 0x2 ref 0 00000100:00100000:2.0:1789760736.794039:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760736.794541:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484217856 00000100:00100000:0.0:1789760736.794555:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484217856 00000100:00100000:0.0:1789760736.794640:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726030640157 time 726073610395 state 0x0 ref 1 00000100:00100000:0.0:1789760736.794660:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726073870811 time 726073610395 state 0x1 ref 1 00000100:00100000:0.0:1789760736.794697:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726073870811 time 726125627040 state 0x0 ref 1 00000100:00100000:0.0:1789760736.794713:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484217856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760736.794788:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20918272, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:0.0:1789760736.794799:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.794821:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:0.0:1789760736.794827:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760736.794834:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20918272->20922367] (req 20918272->20922367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.794861:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20918272->20922367] (req 20918272->20922367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760736.841374:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972425 is committed 00002000:00010000:2.0:1789760736.841619:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20918272 -> 20922368 00010000:00010000:2.0:1789760736.841642:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20918272->20922367] (req 20918272->20922367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.841663:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20918272->20922367] (req 20918272->20922367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.841673:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20918272->20922367] (req 20918272->20922367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.841682:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760736.841688:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20918272->20922367] (req 20918272->20922367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.841699:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20918272->20922367] (req 20918272->20922367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.841724:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20918272->20922367] (req 20918272->20922367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.841732:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208562 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20918272->20922367] (req 20918272->20922367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760736.841992:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484217856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47275us (47961us total) trans 4294972425 rc 0/0 00000100:00100000:2.0:1789760736.842004:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5136 00000100:00100000:2.0:1789760736.842015:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726073870811 time 726125627040 state 0x2 ref 0 00000100:00100000:1.0:1789760736.846311:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760736.846429:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484217984 00000100:00100000:0.0:1789760736.846442:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484217984 00000100:00100000:0.0:1789760736.846590:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726073870811 time 726125627040 state 0x0 ref 1 00000100:00100000:0.0:1789760736.846611:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726125887456 time 726125627040 state 0x1 ref 1 00000100:00100000:0.0:1789760736.846692:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726125887456 time 726177621677 state 0x0 ref 1 00000100:00100000:0.0:1789760736.846710:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484217984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760736.846776:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20922368, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:0.0:1789760736.846787:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.846806:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:0.0:1789760736.846823:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760736.846830:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20922368->20926463] (req 20922368->20926463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.846854:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20922368->20926463] (req 20922368->20926463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760736.890600:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972426 is committed 00002000:00010000:0.0:1789760736.890727:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20922368 -> 20926464 00010000:00010000:0.0:1789760736.890877:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20922368->20926463] (req 20922368->20926463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.890896:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20922368->20926463] (req 20922368->20926463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.890907:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20922368->20926463] (req 20922368->20926463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.890916:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760736.890922:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20922368->20926463] (req 20922368->20926463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.890935:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20922368->20926463] (req 20922368->20926463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.890962:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20922368->20926463] (req 20922368->20926463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.890971:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208569 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20922368->20926463] (req 20922368->20926463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760736.891372:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484217984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44659us (45068us total) trans 4294972426 rc 0/0 00000100:00100000:0.0:1789760736.891384:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5137 00000100:00100000:0.0:1789760736.891396:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726125887456 time 726177621677 state 0x2 ref 0 00000100:00100000:1.0:1789760736.895051:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760736.895297:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484218112 00000100:00100000:2.0:1789760736.895311:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484218112 00000100:00100000:2.0:1789760736.895405:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726125887456 time 726177621677 state 0x0 ref 1 00000100:00100000:2.0:1789760736.895424:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726177882093 time 726177621677 state 0x1 ref 1 00000100:00100000:2.0:1789760736.895464:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726177882093 time 726226394234 state 0x0 ref 1 00000100:00100000:2.0:1789760736.895478:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484218112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760736.895544:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20926464, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:2.0:1789760736.895556:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.895576:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:2.0:1789760736.895581:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760736.895590:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20926464->20930559] (req 20926464->20930559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760736.895607:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20926464->20930559] (req 20926464->20930559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760736.928657:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20926464 -> 20930560 00010000:00010000:1.0:1789760736.928685:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20926464->20930559] (req 20926464->20930559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.928701:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20926464->20930559] (req 20926464->20930559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.928711:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20926464->20930559] (req 20926464->20930559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.928719:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760736.928725:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20926464->20930559] (req 20926464->20930559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.928742:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20926464->20930559] (req 20926464->20930559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.928767:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20926464->20930559] (req 20926464->20930559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760736.928775:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208570 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20926464->20930559] (req 20926464->20930559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760736.928912:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972427 is committed 00000100:00100000:1.0:1789760736.929118:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484218112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33635us (34074us total) trans 4294972427 rc 0/0 00000100:00100000:1.0:1789760736.929142:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5138 00000100:00100000:1.0:1789760736.929155:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726177882093 time 726226394234 state 0x2 ref 0 00000100:00100000:1.0:1789760736.934114:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760736.934237:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484218240 00000100:00100000:3.0:1789760736.934251:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484218240 00000100:00100000:3.0:1789760736.934320:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726177882093 time 726226394234 state 0x0 ref 1 00000100:00100000:3.0:1789760736.934338:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726226654650 time 726226394234 state 0x1 ref 1 00000100:00100000:3.0:1789760736.934374:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726226654650 time 726265304970 state 0x0 ref 1 00000100:00100000:3.0:1789760736.934388:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484218240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760736.934445:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20930560, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:3.0:1789760736.934456:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.934475:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:3.0:1789760736.934481:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760736.934489:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20930560->20934655] (req 20930560->20934655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.934505:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20930560->20934655] (req 20930560->20934655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760736.983122:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972428 is committed 00002000:00010000:3.0:1789760736.983537:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20930560 -> 20934656 00010000:00010000:3.0:1789760736.983556:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20930560->20934655] (req 20930560->20934655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.983571:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20930560->20934655] (req 20930560->20934655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.983581:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20930560->20934655] (req 20930560->20934655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.983588:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760736.983593:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20930560->20934655] (req 20930560->20934655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.983602:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20930560->20934655] (req 20930560->20934655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.983627:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20930560->20934655] (req 20930560->20934655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760736.983635:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208577 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20930560->20934655] (req 20930560->20934655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760736.983758:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484218240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49365us (49687us total) trans 4294972428 rc 0/0 00000100:00100000:3.0:1789760736.983774:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5139 00000100:00100000:3.0:1789760736.983787:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726226654650 time 726265304970 state 0x2 ref 0 00000100:00100000:3.0:1789760736.986862:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760736.986942:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484218368 00000100:00100000:0.0:1789760736.986955:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484218368 00000100:00100000:0.0:1789760736.987041:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726226654650 time 726265304970 state 0x0 ref 1 00000100:00100000:0.0:1789760736.987113:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726265565386 time 726265304970 state 0x1 ref 1 00000100:00100000:0.0:1789760736.987152:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726265565386 time 726318081935 state 0x0 ref 1 00000100:00100000:0.0:1789760736.987169:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484218368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760736.987232:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20934656, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:0.0:1789760736.987244:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.987266:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:0.0:1789760736.987272:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760736.987279:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20934656->20938751] (req 20934656->20938751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760736.987296:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20934656->20938751] (req 20934656->20938751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760737.034247:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20934656 -> 20938752 00010000:00010000:1.0:1789760737.034275:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20934656->20938751] (req 20934656->20938751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.034290:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20934656->20938751] (req 20934656->20938751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.034310:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20934656->20938751] (req 20934656->20938751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.034325:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760737.034332:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20934656->20938751] (req 20934656->20938751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.034342:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20934656->20938751] (req 20934656->20938751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.034378:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20934656->20938751] (req 20934656->20938751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.034386:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620857e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20934656->20938751] (req 20934656->20938751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760737.034525:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484218368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47355us (47673us total) trans 4294972429 rc 0/0 00000100:00100000:1.0:1789760737.034538:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5140 00000100:00100000:1.0:1789760737.034550:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726265565386 time 726318081935 state 0x2 ref 0 00000001:00080000:3.0:1789760737.035558:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972429 is committed 00000100:00100000:1.0:1789760737.037850:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760737.038880:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484218496 00000100:00100000:0.0:1789760737.038897:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484218496 00000100:00100000:0.0:1789760737.039044:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726265565386 time 726318081935 state 0x0 ref 1 00000100:00100000:0.0:1789760737.039110:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726318342351 time 726318081935 state 0x1 ref 1 00000100:00100000:0.0:1789760737.039148:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726318342351 time 726370078631 state 0x0 ref 1 00000100:00100000:0.0:1789760737.039164:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484218496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760737.039231:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20938752, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:0.0:1789760737.039240:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.039260:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:0.0:1789760737.039264:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760737.039272:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20938752->20942847] (req 20938752->20942847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.039289:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20938752->20942847] (req 20938752->20942847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760737.096848:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972430 is committed 00002000:00010000:2.0:1789760737.097025:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20938752 -> 20942848 00010000:00010000:2.0:1789760737.097046:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20938752->20942847] (req 20938752->20942847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.097062:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20938752->20942847] (req 20938752->20942847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.097225:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20938752->20942847] (req 20938752->20942847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.097234:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760737.097240:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20938752->20942847] (req 20938752->20942847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.097252:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20938752->20942847] (req 20938752->20942847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.097278:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20938752->20942847] (req 20938752->20942847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.097287:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208585 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20938752->20942847] (req 20938752->20942847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760737.097763:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484218496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58594us (59918us total) trans 4294972430 rc 0/0 00000100:00100000:3.0:1789760737.097776:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5141 00000100:00100000:3.0:1789760737.097789:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726318342351 time 726370078631 state 0x2 ref 0 00000100:00100000:3.0:1789760737.109614:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.113547:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484218624 00000100:00100000:1.0:1789760737.113573:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484218624 00000100:00100000:0.0:1789760737.117395:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726318342351 time 726370078631 state 0x0 ref 1 00000100:00100000:0.0:1789760737.117996:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726370339047 time 726370078631 state 0x1 ref 1 00000100:00100000:0.0:1789760737.118194:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726370339047 time 726449123464 state 0x0 ref 1 00000100:00100000:0.0:1789760737.118220:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484218624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760737.118300:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20942848, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:0.0:1789760737.118311:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.118330:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:0.0:1789760737.118335:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760737.118343:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20942848->20946943] (req 20942848->20946943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.118374:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20942848->20946943] (req 20942848->20946943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760737.165189:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972431 is committed 00002000:00010000:0.0:1789760737.165405:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20942848 -> 20946944 00010000:00010000:0.0:1789760737.165658:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20942848->20946943] (req 20942848->20946943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.165675:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20942848->20946943] (req 20942848->20946943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.165686:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20942848->20946943] (req 20942848->20946943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.165694:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760737.165701:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20942848->20946943] (req 20942848->20946943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.165712:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20942848->20946943] (req 20942848->20946943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.165738:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20942848->20946943] (req 20942848->20946943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.165746:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620858c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20942848->20946943] (req 20942848->20946943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760737.165920:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484218624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47698us (56309us total) trans 4294972431 rc 0/0 00000100:00100000:0.0:1789760737.165934:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5142 00000100:00100000:0.0:1789760737.165946:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726370339047 time 726449123464 state 0x2 ref 0 00010000:00010000:0.0:1789760737.166422:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760737.170668:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760737.170761:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484218752 00000100:00100000:3.0:1789760737.170774:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484218752 00000100:00100000:3.0:1789760737.170839:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726370339047 time 726449123464 state 0x0 ref 1 00000100:00100000:3.0:1789760737.170857:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726449383880 time 726449123464 state 0x1 ref 1 00000100:00100000:3.0:1789760737.170979:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726449383880 time 726501908834 state 0x0 ref 1 00000100:00100000:3.0:1789760737.170993:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484218752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760737.171063:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20946944, mtime: 0x6aad94e0, atime: 0x0, ctime: 0x6aad94e0, blocks: 0xa007 00010000:00010000:3.0:1789760737.171095:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.171111:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:3.0:1789760737.171116:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760737.171124:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20946944->20951039] (req 20946944->20951039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.171198:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20946944->20951039] (req 20946944->20951039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760737.282601:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972432 is committed 00002000:00010000:3.0:1789760737.282805:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20946944 -> 20951040 00002000:00010000:3.0:1789760737.282811:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760736 -> 1789760737 00002000:00010000:3.0:1789760737.282814:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760736 -> 1789760737 00010000:00010000:3.0:1789760737.282834:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20946944->20951039] (req 20946944->20951039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.282850:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20946944->20951039] (req 20946944->20951039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.282862:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20946944->20951039] (req 20946944->20951039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.282870:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760737.282876:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20946944->20951039] (req 20946944->20951039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.282887:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20946944->20951039] (req 20946944->20951039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.282914:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20946944->20951039] (req 20946944->20951039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.282928:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208593 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20946944->20951039] (req 20946944->20951039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760737.283059:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484218752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 112060us (112394us total) trans 4294972432 rc 0/0 00000100:00100000:3.0:1789760737.283127:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5143 00000100:00100000:3.0:1789760737.283151:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726449383880 time 726501908834 state 0x2 ref 0 00000100:00100000:3.0:1789760737.285971:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760737.286090:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484218880 00000100:00100000:0.0:1789760737.286103:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484218880 00000100:00100000:0.0:1789760737.286191:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726449383880 time 726501908834 state 0x0 ref 1 00000100:00100000:0.0:1789760737.286210:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726502169250 time 726501908834 state 0x1 ref 1 00000100:00100000:0.0:1789760737.286242:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726502169250 time 726617172748 state 0x0 ref 1 00000100:00100000:0.0:1789760737.286257:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484218880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760737.286327:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20951040, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa007 00010000:00010000:0.0:1789760737.286338:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.286369:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:0.0:1789760737.286375:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760737.286382:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20951040->20955135] (req 20951040->20955135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.286400:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20951040->20955135] (req 20951040->20955135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760737.329259:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972433 is committed 00002000:00010000:2.0:1789760737.329274:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20951040 -> 20955136 00010000:00010000:2.0:1789760737.329295:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20951040->20955135] (req 20951040->20955135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.329311:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20951040->20955135] (req 20951040->20955135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.329322:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20951040->20955135] (req 20951040->20955135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.329329:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760737.329334:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20951040->20955135] (req 20951040->20955135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.329341:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20951040->20955135] (req 20951040->20955135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.329372:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20951040->20955135] (req 20951040->20955135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.329378:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620859a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20951040->20955135] (req 20951040->20955135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760737.329537:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484218880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43276us (43575us total) trans 4294972433 rc 0/0 00000100:00100000:2.0:1789760737.329549:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5144 00000100:00100000:2.0:1789760737.329562:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726502169250 time 726617172748 state 0x2 ref 0 00000100:00100000:2.0:1789760737.331961:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760737.332136:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484219008 00000100:00100000:0.0:1789760737.332149:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484219008 00000100:00100000:0.0:1789760737.332230:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726502169250 time 726617172748 state 0x0 ref 1 00000100:00100000:0.0:1789760737.332252:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726617433164 time 726617172748 state 0x1 ref 1 00000100:00100000:3.0:1789760737.332312:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726617433164 time 726663239617 state 0x0 ref 1 00000100:00100000:3.0:1789760737.332334:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484219008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760737.332451:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20955136, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa007 00010000:00010000:3.0:1789760737.332462:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.332482:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:3.0:1789760737.332487:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760737.332495:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20955136->20959231] (req 20955136->20959231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.332514:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20955136->20959231] (req 20955136->20959231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760737.410847:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972434 is committed 00002000:00010000:1.0:1789760737.411489:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20955136 -> 20959232 00010000:00010000:1.0:1789760737.411511:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20955136->20959231] (req 20955136->20959231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.411533:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20955136->20959231] (req 20955136->20959231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.411544:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20955136->20959231] (req 20955136->20959231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.411553:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760737.411559:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20955136->20959231] (req 20955136->20959231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.411571:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20955136->20959231] (req 20955136->20959231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.411597:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20955136->20959231] (req 20955136->20959231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.411607:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962085a1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20955136->20959231] (req 20955136->20959231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760737.411737:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484219008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79405us (79781us total) trans 4294972434 rc 0/0 00000100:00100000:1.0:1789760737.411749:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5145 00000100:00100000:1.0:1789760737.411762:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726617433164 time 726663239617 state 0x2 ref 0 00000100:00100000:1.0:1789760737.415425:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760737.415691:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484219136 00000100:00100000:3.0:1789760737.415706:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484219136 00000100:00100000:3.0:1789760737.415781:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726617433164 time 726663239617 state 0x0 ref 1 00000100:00100000:3.0:1789760737.415802:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726663500033 time 726663239617 state 0x1 ref 1 00000100:00100000:3.0:1789760737.415943:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726663500033 time 726746872571 state 0x0 ref 1 00000100:00100000:3.0:1789760737.415959:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484219136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760737.416029:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20959232, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa007 00010000:00010000:3.0:1789760737.416040:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.416060:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:3.0:1789760737.416095:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760737.416104:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20959232->20963327] (req 20959232->20963327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.416122:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20959232->20963327] (req 20959232->20963327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760737.455722:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20959232 -> 20963328 00010000:00010000:1.0:1789760737.455752:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20959232->20963327] (req 20959232->20963327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.455767:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20959232->20963327] (req 20959232->20963327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.455776:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20959232->20963327] (req 20959232->20963327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.455784:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760737.455789:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20959232->20963327] (req 20959232->20963327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.455800:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20959232->20963327] (req 20959232->20963327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.455824:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20959232->20963327] (req 20959232->20963327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.455832:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962085a8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20959232->20963327] (req 20959232->20963327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760737.455955:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484219136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39991us (40539us total) trans 4294972435 rc 0/0 00000100:00100000:1.0:1789760737.455968:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5146 00000100:00100000:1.0:1789760737.455980:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726663500033 time 726746872571 state 0x2 ref 0 00000001:00080000:2.0:1789760737.456264:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972435 is committed 00000100:00100000:0.0:1789760737.459297:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.460023:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484219264 00000100:00100000:1.0:1789760737.460037:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484219264 00000100:00100000:1.0:1789760737.460152:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726663500033 time 726746872571 state 0x0 ref 1 00000100:00100000:1.0:1789760737.460173:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726747132987 time 726746872571 state 0x1 ref 1 00000100:00100000:1.0:1789760737.460698:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726747132987 time 726791626762 state 0x0 ref 1 00000100:00100000:1.0:1789760737.460716:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484219264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760737.460790:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20963328, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa007 00010000:00010000:1.0:1789760737.460800:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.460820:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:1.0:1789760737.460824:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760737.460832:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20963328->20967423] (req 20963328->20967423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.460848:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20963328->20967423] (req 20963328->20967423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760737.509852:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.509882:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.509918:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.509944:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.509972:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.510005:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484219392 00000100:00100000:1.0:1789760737.510017:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484219392 00000100:00100000:1.0:1789760737.510100:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 477 00000100:00100000:1.0:1789760737.510112:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b6450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484219392:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760737.510253:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b6450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484219392:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 143us (332us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760737.510261:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 477 00000100:00100000:1.0:1789760737.510298:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484219520 00000100:00100000:1.0:1789760737.510302:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484219520 00000100:00100000:1.0:1789760737.510336:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484219776 00000100:00100000:1.0:1789760737.510338:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484219776 00000100:00100000:1.0:1789760737.510366:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484219648 00000100:00100000:1.0:1789760737.510369:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484219648 00000100:00100000:1.0:1789760737.510381:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484219904 00000100:00100000:1.0:1789760737.510383:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484219904 00000100:00100000:1.0:1789760737.510395:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 911 00000100:00100000:1.0:1789760737.510400:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2422:x1876699484219520:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760737.510434:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2422:x1876699484219520:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 33us (596us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760737.510440:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 911 00000100:00100000:1.0:1789760737.510452:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 912 00000100:00100000:1.0:1789760737.510458:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484219776:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760737.510482:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484219776:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 24us (600us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760737.510489:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 912 00000100:00100000:1.0:1789760737.510501:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 913 00000100:00100000:1.0:1789760737.510507:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484219648:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760737.510531:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484219648:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 25us (587us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760737.510537:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 913 00000100:00100000:1.0:1789760737.510548:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 914 00000100:00100000:1.0:1789760737.510554:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484219904:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760737.510577:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484219904:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 23us (604us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760737.510583:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 914 00000100:00080000:1.0:1789760737.511029:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:1.0:1789760737.511035:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000001:00080000:2.0:1789760737.512120:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972436 is committed 00002000:00010000:0.0:1789760737.512592:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20963328 -> 20967424 00010000:00010000:0.0:1789760737.512612:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20963328->20967423] (req 20963328->20967423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.512628:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20963328->20967423] (req 20963328->20967423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.512639:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20963328->20967423] (req 20963328->20967423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.512648:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760737.512654:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20963328->20967423] (req 20963328->20967423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.512665:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20963328->20967423] (req 20963328->20967423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.512690:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20963328->20967423] (req 20963328->20967423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.512699:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962085af lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20963328->20967423] (req 20963328->20967423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760737.512837:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484219264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52118us (53611us total) trans 4294972436 rc 0/0 00000100:00100000:0.0:1789760737.512849:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5147 00000100:00100000:0.0:1789760737.512861:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726747132987 time 726791626762 state 0x2 ref 0 00000100:00100000:1.0:1789760737.516724:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760737.516838:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484220032 00000100:00100000:2.0:1789760737.516852:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484220032 00000100:00100000:2.0:1789760737.516940:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726747132987 time 726791626762 state 0x0 ref 1 00000100:00100000:2.0:1789760737.516962:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726791887178 time 726791626762 state 0x1 ref 1 00000100:00100000:2.0:1789760737.516996:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726791887178 time 726847926605 state 0x0 ref 1 00000100:00100000:2.0:1789760737.517011:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484220032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760737.517144:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20967424, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa007 00010000:00010000:2.0:1789760737.517156:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.517177:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:2.0:1789760737.517183:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760737.517191:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20967424->20971519] (req 20967424->20971519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.517207:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20967424->20971519] (req 20967424->20971519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760737.564495:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972437 is committed 00002000:00010000:2.0:1789760737.564629:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20967424 -> 20971520 00010000:00010000:2.0:1789760737.564648:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20967424->20971519] (req 20967424->20971519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.564660:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20967424->20971519] (req 20967424->20971519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.564666:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20967424->20971519] (req 20967424->20971519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.564685:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760737.564690:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20967424->20971519] (req 20967424->20971519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.564697:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20967424->20971519] (req 20967424->20971519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.564718:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20967424->20971519] (req 20967424->20971519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.564722:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085b6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20967424->20971519] (req 20967424->20971519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760737.564825:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484220032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47811us (48108us total) trans 4294972437 rc 0/0 00000100:00100000:2.0:1789760737.564837:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5148 00000100:00100000:2.0:1789760737.564849:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726791887178 time 726847926605 state 0x2 ref 0 00000100:00100000:3.0:1789760737.567276:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.567427:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484220160 00000100:00100000:1.0:1789760737.567495:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484220160 00000100:00100000:1.0:1789760737.567574:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726791887178 time 726847926605 state 0x0 ref 1 00000100:00100000:1.0:1789760737.567593:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726848187021 time 726847926605 state 0x1 ref 1 00000100:00100000:1.0:1789760737.567683:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726848187021 time 726898612442 state 0x0 ref 1 00000100:00100000:1.0:1789760737.567698:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484220160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760737.567766:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20971520, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa007 00010000:00010000:1.0:1789760737.567776:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.567796:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:1.0:1789760737.567803:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760737.567810:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20971520->20975615] (req 20971520->20975615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.567826:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20971520->20975615] (req 20971520->20975615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760737.621322:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20971520 -> 20975616 00002000:00010000:1.0:1789760737.621337:0:8106:0:(ofd_lvb.c:331:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb blocks from disk: 40967 -> 43015 00010000:00010000:1.0:1789760737.621363:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20971520->20975615] (req 20971520->20975615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.621381:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20971520->20975615] (req 20971520->20975615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.621392:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20971520->20975615] (req 20971520->20975615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.621401:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760737.621406:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20971520->20975615] (req 20971520->20975615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.621416:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20971520->20975615] (req 20971520->20975615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.621541:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20971520->20975615] (req 20971520->20975615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.621558:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085bd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20971520->20975615] (req 20971520->20975615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760737.621683:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484220160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53981us (54413us total) trans 4294972438 rc 0/0 00000100:00100000:1.0:1789760737.621696:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5149 00000100:00100000:1.0:1789760737.621709:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726848187021 time 726898612442 state 0x2 ref 0 00000001:00080000:2.0:1789760737.624061:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972438 is committed 00000100:00100000:1.0:1789760737.624860:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760737.624945:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484220288 00000100:00100000:3.0:1789760737.624958:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484220288 00000100:00100000:3.0:1789760737.625027:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726848187021 time 726898612442 state 0x0 ref 1 00000100:00100000:3.0:1789760737.625047:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726898872858 time 726898612442 state 0x1 ref 1 00000100:00100000:3.0:1789760737.625108:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726898872858 time 726956037925 state 0x0 ref 1 00000100:00100000:3.0:1789760737.625123:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484220288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760737.625201:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20975616, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:3.0:1789760737.625213:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.625233:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=726s v=(0 0) 00010000:00010000:3.0:1789760737.625237:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760737.625245:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20975616->20979711] (req 20975616->20979711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.625263:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20975616->20979711] (req 20975616->20979711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760737.674044:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972439 is committed 00002000:00010000:3.0:1789760737.674222:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20975616 -> 20979712 00010000:00010000:3.0:1789760737.674242:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20975616->20979711] (req 20975616->20979711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.674257:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20975616->20979711] (req 20975616->20979711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.674267:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20975616->20979711] (req 20975616->20979711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.674276:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760737.674281:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20975616->20979711] (req 20975616->20979711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.674291:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20975616->20979711] (req 20975616->20979711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.674318:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20975616->20979711] (req 20975616->20979711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760737.674327:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962085c4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20975616->20979711] (req 20975616->20979711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760737.674461:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484220288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49332us (49600us total) trans 4294972439 rc 0/0 00000100:00100000:3.0:1789760737.674473:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5150 00000100:00100000:3.0:1789760737.674484:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726898872858 time 726956037925 state 0x2 ref 0 00000100:00100000:3.0:1789760737.681256:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.681400:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484220416 00000100:00100000:1.0:1789760737.681414:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484220416 00000100:00100000:1.0:1789760737.681487:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726898872858 time 726956037925 state 0x0 ref 1 00000100:00100000:1.0:1789760737.681505:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726956298341 time 726956037925 state 0x1 ref 1 00000100:00100000:1.0:1789760737.681538:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726956298341 time 727012469111 state 0x0 ref 1 00000100:00100000:1.0:1789760737.681554:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484220416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760737.681619:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20979712, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:1.0:1789760737.681631:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.681651:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760737.681656:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760737.681665:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20979712->20983807] (req 20979712->20983807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.681682:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20979712->20983807] (req 20979712->20983807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760737.718109:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20979712 -> 20983808 00010000:00010000:1.0:1789760737.718208:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20979712->20983807] (req 20979712->20983807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.718225:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20979712->20983807] (req 20979712->20983807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.718236:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20979712->20983807] (req 20979712->20983807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.718245:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760737.718252:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20979712->20983807] (req 20979712->20983807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.718263:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20979712->20983807] (req 20979712->20983807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.718290:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20979712->20983807] (req 20979712->20983807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.718299:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085cb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20979712->20983807] (req 20979712->20983807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760737.718521:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484220416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36963us (37267us total) trans 4294972440 rc 0/0 00000100:00100000:1.0:1789760737.718536:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5151 00000100:00100000:1.0:1789760737.718549:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726956298341 time 727012469111 state 0x2 ref 0 00000001:00080000:1.0:1789760737.718602:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972440 is committed 00000100:00100000:0.0:1789760737.723522:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760737.723704:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484220544 00000100:00100000:2.0:1789760737.723719:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484220544 00000100:00100000:2.0:1789760737.723794:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 726956298341 time 727012469111 state 0x0 ref 1 00000100:00100000:2.0:1789760737.723815:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727012729527 time 727012469111 state 0x1 ref 1 00000100:00100000:2.0:1789760737.724023:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727012729527 time 727054951687 state 0x0 ref 1 00000100:00100000:2.0:1789760737.724041:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484220544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760737.724144:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20983808, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:2.0:1789760737.724156:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.724176:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:2.0:1789760737.724181:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760737.724189:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20983808->20987903] (req 20983808->20987903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.724207:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20983808->20987903] (req 20983808->20987903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760737.772327:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20983808 -> 20987904 00010000:00010000:1.0:1789760737.772359:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20983808->20987903] (req 20983808->20987903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.772372:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20983808->20987903] (req 20983808->20987903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.772383:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20983808->20987903] (req 20983808->20987903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.772391:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760737.772397:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20983808->20987903] (req 20983808->20987903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.772406:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20983808->20987903] (req 20983808->20987903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.772426:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20983808->20987903] (req 20983808->20987903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.772441:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962085d2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20983808->20987903] (req 20983808->20987903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760737.772541:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484220544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48498us (49028us total) trans 4294972441 rc 0/0 00000100:00100000:1.0:1789760737.772553:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5152 00000100:00100000:1.0:1789760737.772564:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727012729527 time 727054951687 state 0x2 ref 0 00000001:00080000:1.0:1789760737.772615:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972441 is committed 00000100:00100000:3.0:1789760737.778475:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.778556:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484220672 00000100:00100000:1.0:1789760737.778571:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484220672 00000100:00100000:1.0:1789760737.778632:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727012729527 time 727054951687 state 0x0 ref 1 00000100:00100000:1.0:1789760737.778646:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727055212103 time 727054951687 state 0x1 ref 1 00000100:00100000:1.0:1789760737.778683:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727055212103 time 727109614634 state 0x0 ref 1 00000100:00100000:1.0:1789760737.778696:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484220672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760737.778740:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20987904, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:1.0:1789760737.778748:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.778764:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760737.778768:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760737.778775:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20987904->20991999] (req 20987904->20991999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.778789:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20987904->20991999] (req 20987904->20991999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760737.828679:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20987904 -> 20992000 00010000:00010000:1.0:1789760737.828713:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20987904->20991999] (req 20987904->20991999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.828729:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20987904->20991999] (req 20987904->20991999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.828818:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20987904->20991999] (req 20987904->20991999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.828827:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760737.828833:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20987904->20991999] (req 20987904->20991999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.828843:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20987904->20991999] (req 20987904->20991999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.828868:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20987904->20991999] (req 20987904->20991999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.828877:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085d9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20987904->20991999] (req 20987904->20991999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760737.829007:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484220672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50307us (50543us total) trans 4294972442 rc 0/0 00000100:00100000:1.0:1789760737.829020:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5153 00000100:00100000:1.0:1789760737.829033:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727055212103 time 727109614634 state 0x2 ref 0 00000001:00080000:1.0:1789760737.829120:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972442 is committed 00000100:00100000:3.0:1789760737.839896:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.840574:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484220800 00000100:00100000:1.0:1789760737.840590:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484220800 00000100:00100000:1.0:1789760737.840669:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727055212103 time 727109614634 state 0x0 ref 1 00000100:00100000:1.0:1789760737.840690:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727109875050 time 727109614634 state 0x1 ref 1 00000100:00100000:1.0:1789760737.840785:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727109875050 time 727171715320 state 0x0 ref 1 00000100:00100000:1.0:1789760737.840801:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484220800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760737.840867:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20992000, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:1.0:1789760737.840878:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.840898:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760737.840903:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760737.840910:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20992000->20996095] (req 20992000->20996095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.840927:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20992000->20996095] (req 20992000->20996095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760737.877933:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20992000 -> 20996096 00010000:00010000:2.0:1789760737.877962:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20992000->20996095] (req 20992000->20996095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.877978:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20992000->20996095] (req 20992000->20996095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.877987:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20992000->20996095] (req 20992000->20996095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.877995:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760737.878000:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20992000->20996095] (req 20992000->20996095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.878011:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20992000->20996095] (req 20992000->20996095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.878036:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20992000->20996095] (req 20992000->20996095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.878044:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962085e0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20992000->20996095] (req 20992000->20996095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760737.878220:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484220800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37415us (38333us total) trans 4294972443 rc 0/0 00000100:00100000:2.0:1789760737.878234:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5154 00000100:00100000:2.0:1789760737.878246:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727109875050 time 727171715320 state 0x2 ref 0 00000001:00080000:1.0:1789760737.878484:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972443 is committed 00000100:00100000:3.0:1789760737.882002:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760737.882120:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484220928 00000100:00100000:0.0:1789760737.882145:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484220928 00000100:00100000:0.0:1789760737.882230:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727109875050 time 727171715320 state 0x0 ref 1 00000100:00100000:0.0:1789760737.882252:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727171975736 time 727171715320 state 0x1 ref 1 00000100:00100000:0.0:1789760737.882288:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727171975736 time 727213218595 state 0x0 ref 1 00000100:00100000:0.0:1789760737.882304:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706f850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484220928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760737.882380:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 20996096, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:0.0:1789760737.882392:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.882412:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:0.0:1789760737.882418:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760737.882426:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20996096->21000191] (req 20996096->21000191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760737.882460:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20996096->21000191] (req 20996096->21000191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760737.928156:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972444 is committed 00002000:00010000:2.0:1789760737.928669:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 20996096 -> 21000192 00010000:00010000:2.0:1789760737.928692:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20996096->21000191] (req 20996096->21000191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.928821:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20996096->21000191] (req 20996096->21000191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.928834:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20996096->21000191] (req 20996096->21000191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.928842:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760737.928849:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20996096->21000191] (req 20996096->21000191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.928860:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20996096->21000191] (req 20996096->21000191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.928887:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20996096->21000191] (req 20996096->21000191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.928895:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962085e7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [20996096->21000191] (req 20996096->21000191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760737.929033:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706f850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484220928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46725us (47039us total) trans 4294972444 rc 0/0 00000100:00100000:2.0:1789760737.929046:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5155 00000100:00100000:2.0:1789760737.929058:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727171975736 time 727213218595 state 0x2 ref 0 00000100:00100000:2.0:1789760737.933248:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760737.933612:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484221056 00000100:00100000:1.0:1789760737.933627:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484221056 00000100:00100000:1.0:1789760737.933706:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727171975736 time 727213218595 state 0x0 ref 1 00000100:00100000:1.0:1789760737.933931:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727213479011 time 727213218595 state 0x1 ref 1 00000100:00100000:1.0:1789760737.934002:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727213479011 time 727264931705 state 0x0 ref 1 00000100:00100000:1.0:1789760737.934018:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484221056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760737.934111:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21000192, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:1.0:1789760737.934123:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.934144:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760737.934149:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760737.934158:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21000192->21004287] (req 21000192->21004287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760737.934175:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21000192->21004287] (req 21000192->21004287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760737.996987:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21000192 -> 21004288 00000001:00080000:3.0:1789760737.996987:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972445 is committed 00010000:00010000:2.0:1789760737.997015:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21000192->21004287] (req 21000192->21004287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.997030:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21000192->21004287] (req 21000192->21004287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.997040:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21000192->21004287] (req 21000192->21004287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.997052:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760737.997057:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21000192->21004287] (req 21000192->21004287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.997096:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21000192->21004287] (req 21000192->21004287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.997118:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21000192->21004287] (req 21000192->21004287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760737.997137:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962085ee lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21000192->21004287] (req 21000192->21004287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760737.997238:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484221056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63220us (64001us total) trans 4294972445 rc 0/0 00000100:00100000:2.0:1789760737.997251:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5156 00000100:00100000:2.0:1789760737.997262:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727213479011 time 727264931705 state 0x2 ref 0 00000100:00100000:3.0:1789760738.000526:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.001877:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484221184 00000100:00100000:1.0:1789760738.001898:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484221184 00000100:00080000:2.0:1789760738.001983:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1789760738.002012:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727213479011 time 727264931705 state 0x0 ref 1 00000100:00100000:1.0:1789760738.002032:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727265192121 time 727264931705 state 0x1 ref 1 00000100:00100000:1.0:1789760738.002117:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727265192121 time 727332995758 state 0x0 ref 1 00000100:00100000:1.0:1789760738.002139:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484221184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.002203:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21004288, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:1.0:1789760738.002213:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.002234:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760738.002239:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.002247:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21004288->21008383] (req 21004288->21008383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.002264:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21004288->21008383] (req 21004288->21008383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760738.049255:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972446 is committed 00002000:00010000:1.0:1789760738.049367:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21004288 -> 21008384 00010000:00010000:1.0:1789760738.049383:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21004288->21008383] (req 21004288->21008383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.049395:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21004288->21008383] (req 21004288->21008383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.049405:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21004288->21008383] (req 21004288->21008383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.049413:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.049418:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21004288->21008383] (req 21004288->21008383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.049436:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21004288->21008383] (req 21004288->21008383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.049455:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21004288->21008383] (req 21004288->21008383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.049462:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962085f5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21004288->21008383] (req 21004288->21008383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.049561:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484221184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47426us (49040us total) trans 4294972446 rc 0/0 00000100:00100000:1.0:1789760738.049572:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5157 00000100:00100000:1.0:1789760738.049580:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727265192121 time 727332995758 state 0x2 ref 0 00000100:00100000:3.0:1789760738.056610:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760738.056702:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484221312 00000100:00100000:2.0:1789760738.056712:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484221312 00000100:00100000:2.0:1789760738.056764:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727265192121 time 727332995758 state 0x0 ref 1 00000100:00100000:2.0:1789760738.056779:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727333256174 time 727332995758 state 0x1 ref 1 00000100:00100000:2.0:1789760738.056826:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727333256174 time 727387750976 state 0x0 ref 1 00000100:00100000:2.0:1789760738.056839:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484221312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760738.056898:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21008384, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:2.0:1789760738.056907:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760738.056923:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:2.0:1789760738.056927:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760738.056933:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21008384->21012479] (req 21008384->21012479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760738.056946:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21008384->21012479] (req 21008384->21012479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760738.094684:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972447 is committed 00002000:00010000:1.0:1789760738.094809:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21008384 -> 21012480 00010000:00010000:1.0:1789760738.094825:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21008384->21012479] (req 21008384->21012479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.094839:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21008384->21012479] (req 21008384->21012479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.094853:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21008384->21012479] (req 21008384->21012479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.094859:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.094864:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21008384->21012479] (req 21008384->21012479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.094873:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21008384->21012479] (req 21008384->21012479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.094891:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21008384->21012479] (req 21008384->21012479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.094898:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962085fc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21008384->21012479] (req 21008384->21012479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.094993:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484221312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38150us (38388us total) trans 4294972447 rc 0/0 00000100:00100000:1.0:1789760738.095003:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5158 00000100:00100000:1.0:1789760738.095012:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727333256174 time 727387750976 state 0x2 ref 0 00000100:00100000:2.0:1789760738.100905:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760738.101379:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484221440 00000100:00100000:0.0:1789760738.101390:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484221440 00000100:00100000:0.0:1789760738.101447:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727333256174 time 727387750976 state 0x0 ref 1 00000100:00100000:0.0:1789760738.101463:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727388011392 time 727387750976 state 0x1 ref 1 00000100:00100000:0.0:1789760738.101523:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727388011392 time 727432453344 state 0x0 ref 1 00000100:00100000:0.0:1789760738.101539:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484221440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760738.101588:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21012480, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:0.0:1789760738.101597:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.101614:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:0.0:1789760738.101617:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760738.101623:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21012480->21016575] (req 21012480->21016575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.101635:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21012480->21016575] (req 21012480->21016575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760738.154650:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972448 is committed 00002000:00010000:1.0:1789760738.154842:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21012480 -> 21016576 00010000:00010000:1.0:1789760738.154857:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21012480->21016575] (req 21012480->21016575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.154875:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21012480->21016575] (req 21012480->21016575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.154885:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21012480->21016575] (req 21012480->21016575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.154893:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.154898:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21012480->21016575] (req 21012480->21016575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.154908:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21012480->21016575] (req 21012480->21016575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.154927:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21012480->21016575] (req 21012480->21016575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.154935:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208603 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21012480->21016575] (req 21012480->21016575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.155035:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484221440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53495us (54249us total) trans 4294972448 rc 0/0 00000100:00100000:1.0:1789760738.155046:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5159 00000100:00100000:1.0:1789760738.155057:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727388011392 time 727432453344 state 0x2 ref 0 00000100:00100000:2.0:1789760738.158679:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.158889:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484221568 00000100:00100000:1.0:1789760738.158900:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484221568 00000100:00100000:1.0:1789760738.158953:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727388011392 time 727432453344 state 0x0 ref 1 00000100:00100000:1.0:1789760738.158969:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727432713760 time 727432453344 state 0x1 ref 1 00000100:00100000:1.0:1789760738.159044:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727432713760 time 727489975105 state 0x0 ref 1 00000100:00100000:1.0:1789760738.159059:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484221568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.159146:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21016576, mtime: 0x6aad94e1, atime: 0x0, ctime: 0x6aad94e1, blocks: 0xa807 00010000:00010000:1.0:1789760738.159156:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.159182:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760738.159187:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.159194:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21016576->21020671] (req 21016576->21020671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.159209:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21016576->21020671] (req 21016576->21020671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760738.188494:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9470380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508919680:0@lo:13:osp-pre-0-1.0 00000100:00100000:2.0:1789760738.188563:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760738.188776:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c4579180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508919808:0@lo:41:osp-pre-0-1.0 00000100:00100000:2.0:1789760738.188820:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760738.189033:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508919808 00000100:00100000:0.0:1789760738.189042:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508919808 00000100:00100000:0.0:1789760738.189114:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 437 00000100:00100000:0.0:1789760738.189137:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508919808:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:0.0:1789760738.189188:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:0.0:1789760738.189238:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018aea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508919808:12345-0@lo:41:osp-pre-0-1.0 Request processed in 115us (417us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760738.189245:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 437 00000100:00100000:2.0:1789760738.189508:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c4579180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508919808:0@lo:41:osp-pre-0-1.0 00000100:00100000:1.0:1789760738.189782:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508919680 00000100:00100000:2.0:1789760738.190119:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:2.0:1789760738.190137:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508919936:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1789760738.190188:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760738.190219:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508919936 00000100:00100000:2.0:1789760738.190224:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508919936 00000100:00100000:2.0:1789760738.190406:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 378 00000100:00100000:2.0:1789760738.190414:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508919936:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:2.0:1789760738.190432:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528154624 pending 1703936 free 3836739584 avail 3825205248 00000100:00100000:2.0:1789760738.190470:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508919936:12345-0@lo:13:osp-pre-0-0.0 Request processed in 57us (281us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760738.190476:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 378 00000100:00080000:2.0:1789760738.190494:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760738.190520:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508919936:0@lo:13:osp-pre-0-0.0 00000100:00100000:1.0:1789760738.190673:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508919680 00000100:00100000:1.0:1789760738.190706:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 379 00000100:00100000:1.0:1789760738.190713:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508919680:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:1.0:1789760738.190724:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528154624 pending 1703936 free 3836739584 avail 3825205248 00000100:00100000:1.0:1789760738.190760:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508919680:12345-0@lo:13:osp-pre-0-1.0 Request processed in 48us (2197us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760738.190769:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 379 00000100:00080000:1.0:1789760738.190783:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1789760738.190889:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9470380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508919680:0@lo:13:osp-pre-0-1.0 00010000:00010000:3.0:1789760738.193033:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760738.203564:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972449 is committed 00002000:00010000:3.0:1789760738.203691:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21016576 -> 21020672 00002000:00010000:3.0:1789760738.203696:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760737 -> 1789760738 00002000:00010000:3.0:1789760738.203698:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760737 -> 1789760738 00010000:00010000:3.0:1789760738.203711:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21016576->21020671] (req 21016576->21020671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.203724:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21016576->21020671] (req 21016576->21020671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.203734:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21016576->21020671] (req 21016576->21020671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.203741:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760738.203746:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21016576->21020671] (req 21016576->21020671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.203755:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21016576->21020671] (req 21016576->21020671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.203774:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21016576->21020671] (req 21016576->21020671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.203782:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620860a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21016576->21020671] (req 21016576->21020671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760738.203870:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484221568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44810us (45197us total) trans 4294972449 rc 0/0 00000100:00100000:3.0:1789760738.203881:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5160 00000100:00100000:3.0:1789760738.203891:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727432713760 time 727489975105 state 0x2 ref 0 00000100:00100000:2.0:1789760738.207373:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.207523:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484221696 00000100:00100000:1.0:1789760738.207533:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484221696 00000100:00100000:1.0:1789760738.207600:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727432713760 time 727489975105 state 0x0 ref 1 00000100:00100000:1.0:1789760738.207623:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727490235521 time 727489975105 state 0x1 ref 1 00000100:00100000:1.0:1789760738.207656:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727490235521 time 727538586780 state 0x0 ref 1 00000100:00100000:1.0:1789760738.207668:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484221696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.207719:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21020672, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:1.0:1789760738.207728:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.207743:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760738.207747:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.207753:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21020672->21024767] (req 21020672->21024767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.207767:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21020672->21024767] (req 21020672->21024767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760738.245154:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972450 is committed 00002000:00010000:1.0:1789760738.245424:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21020672 -> 21024768 00010000:00010000:1.0:1789760738.245439:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21020672->21024767] (req 21020672->21024767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.245453:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21020672->21024767] (req 21020672->21024767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.245461:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21020672->21024767] (req 21020672->21024767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.245468:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.245473:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21020672->21024767] (req 21020672->21024767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.245482:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21020672->21024767] (req 21020672->21024767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.245503:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21020672->21024767] (req 21020672->21024767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.245510:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208611 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21020672->21024767] (req 21020672->21024767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.245638:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484221696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37967us (38269us total) trans 4294972450 rc 0/0 00000100:00100000:1.0:1789760738.245649:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5161 00000100:00100000:1.0:1789760738.245658:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727490235521 time 727538586780 state 0x2 ref 0 00000100:00100000:2.0:1789760738.253659:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.253790:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484221824 00000100:00100000:1.0:1789760738.253801:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484221824 00000100:00100000:1.0:1789760738.253855:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727490235521 time 727538586780 state 0x0 ref 1 00000100:00100000:1.0:1789760738.253869:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727538847196 time 727538586780 state 0x1 ref 1 00000100:00100000:1.0:1789760738.253941:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727538847196 time 727584871907 state 0x0 ref 1 00000100:00100000:1.0:1789760738.253954:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484221824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.253999:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21024768, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:1.0:1789760738.254008:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.254024:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760738.254028:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.254033:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21024768->21028863] (req 21024768->21028863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.254048:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21024768->21028863] (req 21024768->21028863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760738.278940:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972451 is committed 00002000:00010000:1.0:1789760738.279025:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21024768 -> 21028864 00010000:00010000:1.0:1789760738.279040:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21024768->21028863] (req 21024768->21028863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.279054:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21024768->21028863] (req 21024768->21028863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.279088:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21024768->21028863] (req 21024768->21028863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.279096:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.279101:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21024768->21028863] (req 21024768->21028863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.279110:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21024768->21028863] (req 21024768->21028863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.279191:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21024768->21028863] (req 21024768->21028863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.279200:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208618 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21024768->21028863] (req 21024768->21028863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.279318:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484221824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25361us (25664us total) trans 4294972451 rc 0/0 00000100:00100000:1.0:1789760738.279328:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5162 00000100:00100000:1.0:1789760738.279337:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727538847196 time 727584871907 state 0x2 ref 0 00000100:00100000:2.0:1789760738.285881:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760738.286033:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484221952 00000100:00100000:0.0:1789760738.286043:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484221952 00000100:00100000:0.0:1789760738.286134:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727538847196 time 727584871907 state 0x0 ref 1 00000100:00100000:0.0:1789760738.286150:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727585132323 time 727584871907 state 0x1 ref 1 00000100:00100000:0.0:1789760738.286170:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727585132323 time 727617101874 state 0x0 ref 1 00000100:00100000:0.0:1789760738.286184:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484221952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760738.286232:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21028864, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:0.0:1789760738.286241:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.286256:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:0.0:1789760738.286260:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760738.286266:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21028864->21032959] (req 21028864->21032959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.286278:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21028864->21032959] (req 21028864->21032959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760738.344542:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972452 is committed 00002000:00010000:1.0:1789760738.344628:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21028864 -> 21032960 00010000:00010000:1.0:1789760738.344643:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21028864->21032959] (req 21028864->21032959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.344662:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21028864->21032959] (req 21028864->21032959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.344671:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21028864->21032959] (req 21028864->21032959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.344679:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.344683:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21028864->21032959] (req 21028864->21032959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.344692:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21028864->21032959] (req 21028864->21032959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.344712:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21028864->21032959] (req 21028864->21032959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.344720:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620861f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21028864->21032959] (req 21028864->21032959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.344827:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484221952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58642us (58951us total) trans 4294972452 rc 0/0 00000100:00100000:1.0:1789760738.344838:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5163 00000100:00100000:1.0:1789760738.344847:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727585132323 time 727617101874 state 0x2 ref 0 00000100:00100000:2.0:1789760738.348332:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760738.350247:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484222080 00000100:00100000:3.0:1789760738.350261:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484222080 00000100:00100000:3.0:1789760738.350323:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727585132323 time 727617101874 state 0x0 ref 1 00000100:00100000:3.0:1789760738.350348:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727617362290 time 727617101874 state 0x1 ref 1 00000100:00100000:3.0:1789760738.350377:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727617362290 time 727681307950 state 0x0 ref 1 00000100:00100000:3.0:1789760738.350390:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484222080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760738.350438:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21032960, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:3.0:1789760738.350447:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.350462:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:3.0:1789760738.350466:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760738.350473:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21032960->21037055] (req 21032960->21037055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.350486:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21032960->21037055] (req 21032960->21037055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760738.403568:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21032960 -> 21037056 00010000:00010000:3.0:1789760738.403595:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21032960->21037055] (req 21032960->21037055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.403609:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21032960->21037055] (req 21032960->21037055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.403618:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21032960->21037055] (req 21032960->21037055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.403625:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760738.403630:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21032960->21037055] (req 21032960->21037055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.403640:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21032960->21037055] (req 21032960->21037055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.403660:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21032960->21037055] (req 21032960->21037055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.403668:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208626 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21032960->21037055] (req 21032960->21037055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760738.403822:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972453 is committed 00000100:00100000:3.0:1789760738.403928:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484222080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53535us (55600us total) trans 4294972453 rc 0/0 00000100:00100000:3.0:1789760738.403939:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5164 00000100:00100000:3.0:1789760738.403948:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727617362290 time 727681307950 state 0x2 ref 0 00000100:00100000:2.0:1789760738.413564:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.413663:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484222208 00000100:00100000:1.0:1789760738.413673:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484222208 00000100:00100000:1.0:1789760738.413725:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727617362290 time 727681307950 state 0x0 ref 1 00000100:00100000:1.0:1789760738.413752:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727681568366 time 727681307950 state 0x1 ref 1 00000100:00100000:1.0:1789760738.413784:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727681568366 time 727744715190 state 0x0 ref 1 00000100:00100000:1.0:1789760738.413796:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484222208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.413846:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21037056, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:1.0:1789760738.413855:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.413871:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760738.413876:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.413883:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21037056->21041151] (req 21037056->21041151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.413897:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21037056->21041151] (req 21037056->21041151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760738.476138:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21037056 -> 21041152 00010000:00010000:1.0:1789760738.476160:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21037056->21041151] (req 21037056->21041151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.476175:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21037056->21041151] (req 21037056->21041151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.476191:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21037056->21041151] (req 21037056->21041151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.476199:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.476204:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21037056->21041151] (req 21037056->21041151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.476214:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21037056->21041151] (req 21037056->21041151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.476234:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21037056->21041151] (req 21037056->21041151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.476242:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620862d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21037056->21041151] (req 21037056->21041151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.476334:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484222208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62534us (62774us total) trans 4294972454 rc 0/0 00000100:00100000:1.0:1789760738.476354:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5165 00000100:00100000:1.0:1789760738.476364:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727681568366 time 727744715190 state 0x2 ref 0 00000001:00080000:0.0:1789760738.477230:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972454 is committed 00000100:00100000:0.0:1789760738.484210:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.485439:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484222336 00000100:00100000:1.0:1789760738.485452:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484222336 00000100:00100000:1.0:1789760738.485511:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727681568366 time 727744715190 state 0x0 ref 1 00000100:00100000:1.0:1789760738.485527:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727744975606 time 727744715190 state 0x1 ref 1 00000100:00100000:1.0:1789760738.485565:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727744975606 time 727816496289 state 0x0 ref 1 00000100:00100000:1.0:1789760738.485577:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484222336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.485629:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21041152, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:1.0:1789760738.485639:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.485656:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760738.485661:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.485667:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21041152->21045247] (req 21041152->21045247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.485682:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21041152->21045247] (req 21041152->21045247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760738.524997:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972455 is committed 00002000:00010000:1.0:1789760738.525372:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21041152 -> 21045248 00010000:00010000:1.0:1789760738.525389:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21041152->21045247] (req 21041152->21045247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.525402:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21041152->21045247] (req 21041152->21045247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.525411:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21041152->21045247] (req 21041152->21045247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.525417:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.525485:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21041152->21045247] (req 21041152->21045247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.525499:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21041152->21045247] (req 21041152->21045247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.525518:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21041152->21045247] (req 21041152->21045247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.525526:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208634 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21041152->21045247] (req 21041152->21045247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.525904:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484222336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40324us (41699us total) trans 4294972455 rc 0/0 00000100:00100000:1.0:1789760738.525915:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5166 00000100:00100000:1.0:1789760738.525924:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727744975606 time 727816496289 state 0x2 ref 0 00000100:00100000:3.0:1789760738.533290:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760738.535480:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484222464 00000100:00100000:0.0:1789760738.535495:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484222464 00000100:00100000:0.0:1789760738.535561:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727744975606 time 727816496289 state 0x0 ref 1 00000100:00100000:0.0:1789760738.535576:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727816756705 time 727816496289 state 0x1 ref 1 00000100:00100000:0.0:1789760738.535613:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727816756705 time 727866543405 state 0x0 ref 1 00000100:00100000:0.0:1789760738.535625:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484222464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760738.535668:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21045248, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:0.0:1789760738.535680:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.535695:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:0.0:1789760738.535699:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760738.535704:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21045248->21049343] (req 21045248->21049343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.535718:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21045248->21049343] (req 21045248->21049343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760738.587043:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972456 is committed 00002000:00010000:1.0:1789760738.587285:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21045248 -> 21049344 00010000:00010000:1.0:1789760738.587301:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21045248->21049343] (req 21045248->21049343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.587313:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21045248->21049343] (req 21045248->21049343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.587322:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21045248->21049343] (req 21045248->21049343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.587333:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.587349:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21045248->21049343] (req 21045248->21049343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.587359:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21045248->21049343] (req 21045248->21049343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.587378:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21045248->21049343] (req 21045248->21049343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.587385:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620863b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21045248->21049343] (req 21045248->21049343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.587517:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484222464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51890us (54233us total) trans 4294972456 rc 0/0 00000100:00100000:1.0:1789760738.587528:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5167 00000100:00100000:1.0:1789760738.587539:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727816756705 time 727866543405 state 0x2 ref 0 00000100:00100000:2.0:1789760738.596172:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760738.596313:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484222592 00000100:00100000:3.0:1789760738.596323:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484222592 00000100:00100000:3.0:1789760738.596383:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727816756705 time 727866543405 state 0x0 ref 1 00000100:00100000:3.0:1789760738.596399:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727866803821 time 727866543405 state 0x1 ref 1 00000100:00100000:3.0:1789760738.596445:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727866803821 time 727927368843 state 0x0 ref 1 00000100:00100000:3.0:1789760738.596459:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484222592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760738.596506:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21049344, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:3.0:1789760738.596514:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.596530:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:3.0:1789760738.596534:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760738.596542:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21049344->21053439] (req 21049344->21053439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.596556:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21049344->21053439] (req 21049344->21053439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760738.626245:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21049344 -> 21053440 00010000:00010000:1.0:1789760738.626265:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21049344->21053439] (req 21049344->21053439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.626279:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21049344->21053439] (req 21049344->21053439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.626294:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21049344->21053439] (req 21049344->21053439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.626302:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.626307:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21049344->21053439] (req 21049344->21053439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.626315:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21049344->21053439] (req 21049344->21053439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.626346:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21049344->21053439] (req 21049344->21053439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.626354:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208642 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21049344->21053439] (req 21049344->21053439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760738.626518:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972457 is committed 00000100:00100000:1.0:1789760738.626688:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484222592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30227us (30521us total) trans 4294972457 rc 0/0 00000100:00100000:1.0:1789760738.626699:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5168 00000100:00100000:1.0:1789760738.626708:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727866803821 time 727927368843 state 0x2 ref 0 00000100:00100000:0.0:1789760738.631861:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.633129:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484222720 00000100:00100000:1.0:1789760738.633142:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484222720 00000100:00100000:1.0:1789760738.633201:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727866803821 time 727927368843 state 0x0 ref 1 00000100:00100000:1.0:1789760738.633217:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727927629259 time 727927368843 state 0x1 ref 1 00000100:00100000:1.0:1789760738.633291:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727927629259 time 727964221289 state 0x0 ref 1 00000100:00100000:1.0:1789760738.633304:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484222720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.633363:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21053440, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:1.0:1789760738.633372:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.633397:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=727s v=(0 0) 00010000:00010000:1.0:1789760738.633403:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.633410:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21053440->21057535] (req 21053440->21057535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.633424:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21053440->21057535] (req 21053440->21057535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760738.677751:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972458 is committed 00002000:00010000:1.0:1789760738.677851:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21053440 -> 21057536 00010000:00010000:1.0:1789760738.677868:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21053440->21057535] (req 21053440->21057535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.677881:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21053440->21057535] (req 21053440->21057535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.677891:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21053440->21057535] (req 21053440->21057535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.677900:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.677905:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21053440->21057535] (req 21053440->21057535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.677916:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21053440->21057535] (req 21053440->21057535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.677940:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21053440->21057535] (req 21053440->21057535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.677948:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208649 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21053440->21057535] (req 21053440->21057535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.678042:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484222720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44735us (46184us total) trans 4294972458 rc 0/0 00000100:00100000:1.0:1789760738.678052:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5169 00000100:00100000:1.0:1789760738.678062:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727927629259 time 727964221289 state 0x2 ref 0 00000100:00100000:3.0:1789760738.689395:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.689507:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484222848 00000100:00100000:1.0:1789760738.689664:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484222848 00000100:00100000:1.0:1789760738.689723:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727927629259 time 727964221289 state 0x0 ref 1 00000100:00100000:1.0:1789760738.689739:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727964481705 time 727964221289 state 0x1 ref 1 00000100:00100000:1.0:1789760738.689772:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727964481705 time 728020702809 state 0x0 ref 1 00000100:00100000:1.0:1789760738.689786:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484222848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.689899:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21057536, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:1.0:1789760738.689909:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.689925:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:1.0:1789760738.689930:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.689936:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21057536->21061631] (req 21057536->21061631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.689950:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21057536->21061631] (req 21057536->21061631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760738.702044:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c45c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508920064:0@lo:41:osp-pre-1-0.0 00000100:00100000:0.0:1789760738.702133:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760738.702290:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508920064 00000100:00100000:3.0:1789760738.702299:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508920064 00000100:00100000:3.0:1789760738.702365:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 438 00000100:00100000:3.0:1789760738.702375:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457aa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508920064:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:3.0:1789760738.702414:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:3.0:1789760738.702475:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457aa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508920064:12345-0@lo:41:osp-pre-1-0.0 Request processed in 100us (340us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760738.702483:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 438 00000100:00100000:1.0:1789760738.702565:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c45c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508920064:0@lo:41:osp-pre-1-0.0 00002000:00010000:1.0:1789760738.736785:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21057536 -> 21061632 00010000:00010000:1.0:1789760738.736806:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21057536->21061631] (req 21057536->21061631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.736822:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21057536->21061631] (req 21057536->21061631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.736831:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21057536->21061631] (req 21057536->21061631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.736838:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.736844:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21057536->21061631] (req 21057536->21061631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.736853:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21057536->21061631] (req 21057536->21061631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.736874:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21057536->21061631] (req 21057536->21061631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.736881:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208650 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21057536->21061631] (req 21057536->21061631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.737026:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484222848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47236us (47633us total) trans 4294972459 rc 0/0 00000100:00100000:1.0:1789760738.737037:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5170 00000100:00100000:1.0:1789760738.737048:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727964481705 time 728020702809 state 0x2 ref 0 00000001:00080000:2.0:1789760738.737332:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972459 is committed 00000100:00100000:2.0:1789760738.747173:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760738.747278:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484222976 00000100:00100000:0.0:1789760738.747289:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484222976 00000100:00100000:0.0:1789760738.747357:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 727964481705 time 728020702809 state 0x0 ref 1 00000100:00100000:0.0:1789760738.747373:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728020963225 time 728020702809 state 0x1 ref 1 00000100:00100000:0.0:1789760738.747451:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728020963225 time 728078381450 state 0x0 ref 1 00000100:00100000:0.0:1789760738.747467:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484222976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760738.747515:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21061632, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:0.0:1789760738.747525:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.747546:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:0.0:1789760738.747552:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760738.747558:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21061632->21065727] (req 21061632->21065727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.747572:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21061632->21065727] (req 21061632->21065727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760738.777411:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972460 is committed 00002000:00010000:0.0:1789760738.777499:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21061632 -> 21065728 00010000:00010000:0.0:1789760738.777515:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21061632->21065727] (req 21061632->21065727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.777529:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21061632->21065727] (req 21061632->21065727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.777538:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21061632->21065727] (req 21061632->21065727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.777545:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760738.777549:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21061632->21065727] (req 21061632->21065727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.777558:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21061632->21065727] (req 21061632->21065727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.777576:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21061632->21065727] (req 21061632->21065727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.777584:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208657 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21061632->21065727] (req 21061632->21065727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760738.777790:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484222976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30323us (30721us total) trans 4294972460 rc 0/0 00000100:00100000:0.0:1789760738.777802:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5171 00000100:00100000:0.0:1789760738.777811:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728020963225 time 728078381450 state 0x2 ref 0 00000100:00100000:0.0:1789760738.782480:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760738.782644:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484223104 00000100:00100000:3.0:1789760738.782654:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484223104 00000100:00100000:3.0:1789760738.782722:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728020963225 time 728078381450 state 0x0 ref 1 00000100:00100000:3.0:1789760738.782737:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728078641866 time 728078381450 state 0x1 ref 1 00000100:00100000:3.0:1789760738.782768:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728078641866 time 728113698258 state 0x0 ref 1 00000100:00100000:3.0:1789760738.782780:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484223104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760738.782827:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21065728, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:3.0:1789760738.782848:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.782864:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:3.0:1789760738.782868:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760738.782874:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21065728->21069823] (req 21065728->21069823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.782889:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21065728->21069823] (req 21065728->21069823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760738.810544:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972461 is committed 00002000:00010000:3.0:1789760738.810859:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21065728 -> 21069824 00010000:00010000:3.0:1789760738.810874:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21065728->21069823] (req 21065728->21069823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.810887:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21065728->21069823] (req 21065728->21069823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.810895:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21065728->21069823] (req 21065728->21069823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.810901:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760738.810906:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21065728->21069823] (req 21065728->21069823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.810914:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21065728->21069823] (req 21065728->21069823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.810934:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21065728->21069823] (req 21065728->21069823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760738.810941:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620865e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21065728->21069823] (req 21065728->21069823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760738.811039:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484223104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28256us (28565us total) trans 4294972461 rc 0/0 00000100:00100000:3.0:1789760738.811050:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5172 00000100:00100000:3.0:1789760738.811060:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728078641866 time 728113698258 state 0x2 ref 0 00000100:00100000:3.0:1789760738.819261:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.821467:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484223232 00000100:00100000:1.0:1789760738.821481:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484223232 00000100:00100000:1.0:1789760738.821541:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728078641866 time 728113698258 state 0x0 ref 1 00000100:00100000:1.0:1789760738.821556:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728113958674 time 728113698258 state 0x1 ref 1 00000100:00100000:1.0:1789760738.821637:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728113958674 time 728152567831 state 0x0 ref 1 00000100:00100000:1.0:1789760738.821651:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484223232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.821722:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21069824, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:1.0:1789760738.821731:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.821753:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:1.0:1789760738.821758:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.821765:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21069824->21073919] (req 21069824->21073919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.821779:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21069824->21073919] (req 21069824->21073919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760738.850454:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21069824 -> 21073920 00010000:00010000:1.0:1789760738.850477:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21069824->21073919] (req 21069824->21073919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.850491:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21069824->21073919] (req 21069824->21073919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.850500:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21069824->21073919] (req 21069824->21073919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.850507:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.850512:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21069824->21073919] (req 21069824->21073919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.850522:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21069824->21073919] (req 21069824->21073919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.850542:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21069824->21073919] (req 21069824->21073919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.850550:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208665 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21069824->21073919] (req 21069824->21073919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.850644:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484223232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28991us (31388us total) trans 4294972462 rc 0/0 00000100:00100000:1.0:1789760738.850655:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5173 00000100:00100000:1.0:1789760738.850665:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728113958674 time 728152567831 state 0x2 ref 0 00000001:00080000:3.0:1789760738.851889:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972462 is committed 00000100:00100000:0.0:1789760738.858809:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.859061:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484223360 00000100:00100000:1.0:1789760738.859106:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484223360 00000100:00100000:0.0:1789760738.859292:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728113958674 time 728152567831 state 0x0 ref 1 00000100:00100000:0.0:1789760738.859307:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728152828247 time 728152567831 state 0x1 ref 1 00000100:00100000:0.0:1789760738.859330:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728152828247 time 728190261823 state 0x0 ref 1 00000100:00100000:0.0:1789760738.859359:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484223360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760738.859402:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21073920, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:0.0:1789760738.859411:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.859427:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:0.0:1789760738.859431:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760738.859437:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21073920->21078015] (req 21073920->21078015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.859451:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21073920->21078015] (req 21073920->21078015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760738.894441:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21073920 -> 21078016 00010000:00010000:0.0:1789760738.894460:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21073920->21078015] (req 21073920->21078015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.894473:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21073920->21078015] (req 21073920->21078015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.894480:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21073920->21078015] (req 21073920->21078015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.894487:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760738.894493:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21073920->21078015] (req 21073920->21078015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.894501:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21073920->21078015] (req 21073920->21078015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.894520:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21073920->21078015] (req 21073920->21078015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.894527:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620866c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21073920->21078015] (req 21073920->21078015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760738.894623:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484223360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35268us (35818us total) trans 4294972463 rc 0/0 00000100:00100000:0.0:1789760738.894633:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5174 00000100:00100000:0.0:1789760738.894642:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728152828247 time 728190261823 state 0x2 ref 0 00000001:00080000:1.0:1789760738.895346:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972463 is committed 00000100:00100000:0.0:1789760738.901822:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760738.901933:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484223488 00000100:00100000:1.0:1789760738.901942:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484223488 00000100:00100000:1.0:1789760738.901991:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728152828247 time 728190261823 state 0x0 ref 1 00000100:00100000:1.0:1789760738.902005:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728190522239 time 728190261823 state 0x1 ref 1 00000100:00100000:1.0:1789760738.902038:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728190522239 time 728232968660 state 0x0 ref 1 00000100:00100000:1.0:1789760738.902049:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484223488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760738.902125:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21078016, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:1.0:1789760738.902134:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.902150:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:1.0:1789760738.902155:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760738.902161:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21078016->21082111] (req 21078016->21082111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.902174:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21078016->21082111] (req 21078016->21082111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760738.941540:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972464 is committed 00002000:00010000:1.0:1789760738.941785:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21078016 -> 21082112 00010000:00010000:1.0:1789760738.941801:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21078016->21082111] (req 21078016->21082111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.941894:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21078016->21082111] (req 21078016->21082111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.941907:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21078016->21082111] (req 21078016->21082111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.941914:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.941919:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21078016->21082111] (req 21078016->21082111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.941929:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21078016->21082111] (req 21078016->21082111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.941948:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21078016->21082111] (req 21078016->21082111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.941956:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208673 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21078016->21082111] (req 21078016->21082111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.942048:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484223488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39996us (40232us total) trans 4294972464 rc 0/0 00000100:00100000:1.0:1789760738.942057:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5175 00000100:00100000:1.0:1789760738.942397:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728190522239 time 728232968660 state 0x2 ref 0 00000100:00100000:3.0:1789760738.945402:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760738.945523:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484223616 00000100:00100000:0.0:1789760738.945533:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484223616 00000100:00100000:0.0:1789760738.945597:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728190522239 time 728232968660 state 0x0 ref 1 00000100:00100000:0.0:1789760738.945613:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728233229076 time 728232968660 state 0x1 ref 1 00000100:00100000:0.0:1789760738.945688:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728233229076 time 728276618355 state 0x0 ref 1 00000100:00100000:0.0:1789760738.945703:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484223616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760738.945755:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21082112, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:0.0:1789760738.945764:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.945779:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:0.0:1789760738.945784:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760738.945790:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21082112->21086207] (req 21082112->21086207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760738.945810:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21082112->21086207] (req 21082112->21086207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760738.990093:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972465 is committed 00002000:00010000:1.0:1789760738.990191:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21082112 -> 21086208 00010000:00010000:1.0:1789760738.990207:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21082112->21086207] (req 21082112->21086207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.990220:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21082112->21086207] (req 21082112->21086207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.990228:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21082112->21086207] (req 21082112->21086207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.990236:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760738.990240:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21082112->21086207] (req 21082112->21086207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.990248:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21082112->21086207] (req 21082112->21086207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.990267:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21082112->21086207] (req 21082112->21086207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760738.990280:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620867a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21082112->21086207] (req 21082112->21086207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760738.990575:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484223616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44869us (45177us total) trans 4294972465 rc 0/0 00000100:00100000:1.0:1789760738.990585:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5176 00000100:00100000:1.0:1789760738.990594:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728233229076 time 728276618355 state 0x2 ref 0 00000100:00100000:0.0:1789760738.994379:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760738.994590:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484223744 00000100:00100000:2.0:1789760738.994600:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484223744 00000100:00100000:2.0:1789760738.994652:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728233229076 time 728276618355 state 0x0 ref 1 00000100:00100000:2.0:1789760738.994667:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728276878771 time 728276618355 state 0x1 ref 1 00000100:00100000:2.0:1789760738.994698:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728276878771 time 728325629379 state 0x0 ref 1 00000100:00100000:2.0:1789760738.994712:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484223744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760738.994762:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21086208, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:2.0:1789760738.994770:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760738.994785:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:2.0:1789760738.994790:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760738.994796:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21086208->21090303] (req 21086208->21090303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760738.994810:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21086208->21090303] (req 21086208->21090303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760739.028899:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972466 is committed 00002000:00010000:2.0:1789760739.029013:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21086208 -> 21090304 00010000:00010000:2.0:1789760739.029029:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21086208->21090303] (req 21086208->21090303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.029042:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21086208->21090303] (req 21086208->21090303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.029057:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21086208->21090303] (req 21086208->21090303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.029086:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760739.029092:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21086208->21090303] (req 21086208->21090303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.029101:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21086208->21090303] (req 21086208->21090303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.029196:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21086208->21090303] (req 21086208->21090303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.029205:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208681 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21086208->21090303] (req 21086208->21090303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760739.029355:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484223744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34624us (34963us total) trans 4294972466 rc 0/0 00000100:00100000:2.0:1789760739.029367:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5177 00000100:00100000:2.0:1789760739.029377:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728276878771 time 728325629379 state 0x2 ref 0 00000100:00100000:2.0:1789760739.034920:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.034989:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484223872 00000100:00100000:3.0:1789760739.035000:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484223872 00000100:00100000:3.0:1789760739.035049:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728276878771 time 728325629379 state 0x0 ref 1 00000100:00100000:3.0:1789760739.035095:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728325889795 time 728325629379 state 0x1 ref 1 00000100:00100000:3.0:1789760739.035128:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728325889795 time 728366050485 state 0x0 ref 1 00000100:00100000:3.0:1789760739.035143:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484223872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.035203:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21090304, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:3.0:1789760739.035213:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.035240:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:3.0:1789760739.035245:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.035251:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21090304->21094399] (req 21090304->21094399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.035265:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21090304->21094399] (req 21090304->21094399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760739.066634:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972467 is committed 00002000:00010000:3.0:1789760739.066734:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21090304 -> 21094400 00010000:00010000:3.0:1789760739.066751:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21090304->21094399] (req 21090304->21094399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.066765:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21090304->21094399] (req 21090304->21094399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.066774:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21090304->21094399] (req 21090304->21094399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.066782:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760739.066787:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21090304->21094399] (req 21090304->21094399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.066795:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21090304->21094399] (req 21090304->21094399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.066815:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21090304->21094399] (req 21090304->21094399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.066823:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208688 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21090304->21094399] (req 21090304->21094399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760739.066923:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484223872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31779us (32008us total) trans 4294972467 rc 0/0 00000100:00100000:3.0:1789760739.066933:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5178 00000100:00100000:3.0:1789760739.066942:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728325889795 time 728366050485 state 0x2 ref 0 00000100:00100000:0.0:1789760739.071126:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.071287:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484224000 00000100:00100000:3.0:1789760739.071297:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484224000 00000100:00100000:3.0:1789760739.071369:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728325889795 time 728366050485 state 0x0 ref 1 00000100:00100000:3.0:1789760739.071385:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728366310901 time 728366050485 state 0x1 ref 1 00000100:00100000:3.0:1789760739.071410:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728366310901 time 728402341275 state 0x0 ref 1 00000100:00100000:3.0:1789760739.071422:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484224000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.071489:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21094400, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:3.0:1789760739.071499:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.071517:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:3.0:1789760739.071521:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.071528:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21094400->21098495] (req 21094400->21098495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.071544:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21094400->21098495] (req 21094400->21098495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760739.104962:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21094400 -> 21098496 00010000:00010000:3.0:1789760739.104983:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21094400->21098495] (req 21094400->21098495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.104996:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21094400->21098495] (req 21094400->21098495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.105010:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21094400->21098495] (req 21094400->21098495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.105017:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760739.105022:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21094400->21098495] (req 21094400->21098495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.105032:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21094400->21098495] (req 21094400->21098495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.105052:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21094400->21098495] (req 21094400->21098495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.105061:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620868f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21094400->21098495] (req 21094400->21098495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760739.105268:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484224000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33843us (34153us total) trans 4294972468 rc 0/0 00000100:00100000:3.0:1789760739.105281:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5179 00000100:00100000:3.0:1789760739.105291:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728366310901 time 728402341275 state 0x2 ref 0 00000001:00080000:2.0:1789760739.109086:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972468 is committed 00000100:00100000:0.0:1789760739.109793:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760739.109904:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484224128 00000100:00100000:1.0:1789760739.109914:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484224128 00000100:00100000:1.0:1789760739.109970:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728366310901 time 728402341275 state 0x0 ref 1 00000100:00100000:1.0:1789760739.109985:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728402601691 time 728402341275 state 0x1 ref 1 00000100:00100000:1.0:1789760739.110047:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728402601691 time 728440977674 state 0x0 ref 1 00000100:00100000:1.0:1789760739.110060:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484224128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760739.110140:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21098496, mtime: 0x6aad94e2, atime: 0x0, ctime: 0x6aad94e2, blocks: 0xa807 00010000:00010000:1.0:1789760739.110149:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.110167:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:1.0:1789760739.110171:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760739.110180:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21098496->21102591] (req 21098496->21102591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.110195:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21098496->21102591] (req 21098496->21102591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760739.147164:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972469 is committed 00002000:00010000:2.0:1789760739.147340:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21098496 -> 21102592 00002000:00010000:2.0:1789760739.147346:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760738 -> 1789760739 00002000:00010000:2.0:1789760739.147349:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760738 -> 1789760739 00010000:00010000:2.0:1789760739.147362:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21098496->21102591] (req 21098496->21102591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.147375:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21098496->21102591] (req 21098496->21102591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.147384:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21098496->21102591] (req 21098496->21102591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.147393:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760739.147397:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21098496->21102591] (req 21098496->21102591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.147407:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21098496->21102591] (req 21098496->21102591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.147500:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21098496->21102591] (req 21098496->21102591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.147509:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208696 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21098496->21102591] (req 21098496->21102591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760739.147605:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484224128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37542us (37811us total) trans 4294972469 rc 0/0 00000100:00100000:2.0:1789760739.147615:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5180 00000100:00100000:2.0:1789760739.147625:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728402601691 time 728440977674 state 0x2 ref 0 00000100:00100000:0.0:1789760739.151509:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760739.153640:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484224256 00000100:00100000:2.0:1789760739.153654:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484224256 00000100:00100000:2.0:1789760739.153716:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728402601691 time 728440977674 state 0x0 ref 1 00000100:00100000:2.0:1789760739.153741:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728441238090 time 728440977674 state 0x1 ref 1 00000100:00100000:2.0:1789760739.153773:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728441238090 time 728484704337 state 0x0 ref 1 00000100:00100000:2.0:1789760739.153787:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484224256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760739.153835:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21102592, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:2.0:1789760739.153844:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.153860:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:2.0:1789760739.153865:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760739.153872:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21102592->21106687] (req 21102592->21106687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.153887:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21102592->21106687] (req 21102592->21106687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760739.193409:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972470 is committed 00002000:00010000:2.0:1789760739.193717:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21102592 -> 21106688 00010000:00010000:2.0:1789760739.193732:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21102592->21106687] (req 21102592->21106687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.193745:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21102592->21106687] (req 21102592->21106687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.193754:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21102592->21106687] (req 21102592->21106687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.193761:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760739.193766:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21102592->21106687] (req 21102592->21106687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.193775:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21102592->21106687] (req 21102592->21106687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.193794:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21102592->21106687] (req 21102592->21106687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.193803:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620869d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21102592->21106687] (req 21102592->21106687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760739.193938:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484224256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40150us (42434us total) trans 4294972470 rc 0/0 00000100:00100000:2.0:1789760739.193963:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5181 00000100:00100000:2.0:1789760739.193974:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728441238090 time 728484704337 state 0x2 ref 0 00000100:00100000:0.0:1789760739.197548:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.197633:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484224384 00000100:00100000:3.0:1789760739.197643:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484224384 00000100:00100000:3.0:1789760739.197697:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728441238090 time 728484704337 state 0x0 ref 1 00000100:00100000:3.0:1789760739.197713:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728484964753 time 728484704337 state 0x1 ref 1 00000100:00100000:3.0:1789760739.197744:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728484964753 time 728528674248 state 0x0 ref 1 00000100:00100000:3.0:1789760739.197756:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484224384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.197803:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21106688, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760739.197812:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.197828:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:3.0:1789760739.197833:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.197839:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21106688->21110783] (req 21106688->21110783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.197854:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21106688->21110783] (req 21106688->21110783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.213375:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760739.224057:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972471 is committed 00002000:00010000:3.0:1789760739.224243:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21106688 -> 21110784 00010000:00010000:3.0:1789760739.224258:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21106688->21110783] (req 21106688->21110783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.224270:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21106688->21110783] (req 21106688->21110783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.224280:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21106688->21110783] (req 21106688->21110783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.224288:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760739.224293:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21106688->21110783] (req 21106688->21110783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.224302:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21106688->21110783] (req 21106688->21110783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.224320:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21106688->21110783] (req 21106688->21110783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.224328:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086a4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21106688->21110783] (req 21106688->21110783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760739.224424:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484224384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26665us (26879us total) trans 4294972471 rc 0/0 00000100:00100000:3.0:1789760739.224433:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5182 00000100:00100000:3.0:1789760739.224443:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728484964753 time 728528674248 state 0x2 ref 0 00000100:00100000:1.0:1789760739.227922:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.228020:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484224512 00000100:00100000:3.0:1789760739.228030:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484224512 00000100:00100000:3.0:1789760739.228203:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728484964753 time 728528674248 state 0x0 ref 1 00000100:00100000:3.0:1789760739.228217:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728528934664 time 728528674248 state 0x1 ref 1 00000100:00100000:3.0:1789760739.228509:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728528934664 time 728559437963 state 0x0 ref 1 00000100:00100000:3.0:1789760739.228526:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484224512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.228571:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21110784, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760739.228579:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.228594:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:3.0:1789760739.228597:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.228603:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21110784->21114879] (req 21110784->21114879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.228616:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21110784->21114879] (req 21110784->21114879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760739.268150:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972472 is committed 00002000:00010000:3.0:1789760739.268316:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21110784 -> 21114880 00010000:00010000:3.0:1789760739.268342:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21110784->21114879] (req 21110784->21114879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.268357:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21110784->21114879] (req 21110784->21114879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.268368:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21110784->21114879] (req 21110784->21114879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.268386:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760739.268391:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21110784->21114879] (req 21110784->21114879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.268401:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21110784->21114879] (req 21110784->21114879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.268547:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21110784->21114879] (req 21110784->21114879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.268556:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086ab lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21110784->21114879] (req 21110784->21114879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760739.268656:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484224512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40129us (40738us total) trans 4294972472 rc 0/0 00000100:00100000:3.0:1789760739.268666:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5183 00000100:00100000:3.0:1789760739.268675:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728528934664 time 728559437963 state 0x2 ref 0 00000100:00100000:3.0:1789760739.272536:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760739.272613:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484224640 00000100:00100000:1.0:1789760739.272625:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484224640 00000100:00100000:1.0:1789760739.272683:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728528934664 time 728559437963 state 0x0 ref 1 00000100:00100000:1.0:1789760739.272703:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728559698379 time 728559437963 state 0x1 ref 1 00000100:00100000:1.0:1789760739.272726:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728559698379 time 728603657152 state 0x0 ref 1 00000100:00100000:1.0:1789760739.272738:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484224640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760739.272857:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21114880, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:1.0:1789760739.272866:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.272882:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:1.0:1789760739.272886:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760739.272892:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21114880->21118975] (req 21114880->21118975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.272906:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21114880->21118975] (req 21114880->21118975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760739.294095:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972473 is committed 00002000:00010000:1.0:1789760739.294320:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21114880 -> 21118976 00010000:00010000:1.0:1789760739.294350:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21114880->21118975] (req 21114880->21118975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.294366:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21114880->21118975] (req 21114880->21118975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.294496:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21114880->21118975] (req 21114880->21118975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.294504:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760739.294509:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21114880->21118975] (req 21114880->21118975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.294519:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21114880->21118975] (req 21114880->21118975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.294539:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21114880->21118975] (req 21114880->21118975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.294548:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086b2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21114880->21118975] (req 21114880->21118975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760739.294648:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484224640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 21906us (22122us total) trans 4294972473 rc 0/0 00000100:00100000:1.0:1789760739.294663:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5184 00000100:00100000:1.0:1789760739.294673:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728559698379 time 728603657152 state 0x2 ref 0 00000100:00100000:2.0:1789760739.298916:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760739.299259:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484224768 00000100:00100000:0.0:1789760739.299270:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484224768 00000100:00100000:1.0:1789760739.299514:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728559698379 time 728603657152 state 0x0 ref 1 00000100:00100000:1.0:1789760739.299531:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728603917568 time 728603657152 state 0x1 ref 1 00000100:00100000:1.0:1789760739.299602:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728603917568 time 728630532119 state 0x0 ref 1 00000100:00100000:1.0:1789760739.299617:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484224768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760739.299665:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21118976, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:1.0:1789760739.299675:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.299700:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:1.0:1789760739.299705:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760739.299712:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21118976->21123071] (req 21118976->21123071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.299727:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21118976->21123071] (req 21118976->21123071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760739.323691:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972474 is committed 00002000:00010000:0.0:1789760739.324290:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21118976 -> 21123072 00010000:00010000:0.0:1789760739.324307:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21118976->21123071] (req 21118976->21123071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.324321:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21118976->21123071] (req 21118976->21123071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.324339:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21118976->21123071] (req 21118976->21123071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.324347:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760739.324352:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21118976->21123071] (req 21118976->21123071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.324361:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21118976->21123071] (req 21118976->21123071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.324387:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21118976->21123071] (req 21118976->21123071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.324394:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962086b9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21118976->21123071] (req 21118976->21123071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760739.324488:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484224768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24870us (25578us total) trans 4294972474 rc 0/0 00000100:00100000:0.0:1789760739.324499:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5185 00000100:00100000:0.0:1789760739.324510:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728603917568 time 728630532119 state 0x2 ref 0 00000100:00100000:2.0:1789760739.328530:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.328622:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484224896 00000100:00100000:3.0:1789760739.328632:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484224896 00000100:00100000:3.0:1789760739.328687:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728603917568 time 728630532119 state 0x0 ref 1 00000100:00100000:3.0:1789760739.328703:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728630792535 time 728630532119 state 0x1 ref 1 00000100:00100000:3.0:1789760739.328736:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728630792535 time 728659666981 state 0x0 ref 1 00000100:00100000:3.0:1789760739.328750:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484224896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.328798:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21123072, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760739.328807:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.328824:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:3.0:1789760739.328828:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.328834:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21123072->21127167] (req 21123072->21127167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.328847:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21123072->21127167] (req 21123072->21127167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760739.353714:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21123072 -> 21127168 00010000:00010000:2.0:1789760739.353735:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21123072->21127167] (req 21123072->21127167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.353749:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21123072->21127167] (req 21123072->21127167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.353759:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21123072->21127167] (req 21123072->21127167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.353767:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760739.353772:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21123072->21127167] (req 21123072->21127167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.353781:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21123072->21127167] (req 21123072->21127167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.353801:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21123072->21127167] (req 21123072->21127167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.353809:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962086c0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21123072->21127167] (req 21123072->21127167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760739.353907:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484224896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25154us (25381us total) trans 4294972475 rc 0/0 00000100:00100000:2.0:1789760739.353918:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5186 00000100:00100000:2.0:1789760739.353928:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728630792535 time 728659666981 state 0x2 ref 0 00000001:00080000:3.0:1789760739.354459:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972475 is committed 00000100:00100000:3.0:1789760739.358125:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760739.358216:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484225024 00000100:00100000:0.0:1789760739.358225:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484225024 00000100:00100000:0.0:1789760739.358287:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728630792535 time 728659666981 state 0x0 ref 1 00000100:00100000:0.0:1789760739.358302:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728659927397 time 728659666981 state 0x1 ref 1 00000100:00100000:0.0:1789760739.358347:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728659927397 time 728689277370 state 0x0 ref 1 00000100:00100000:0.0:1789760739.358361:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484225024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760739.358412:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21127168, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:0.0:1789760739.358421:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.358437:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:0.0:1789760739.358442:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760739.358448:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21127168->21131263] (req 21127168->21131263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.358462:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21127168->21131263] (req 21127168->21131263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760739.393643:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972476 is committed 00002000:00010000:0.0:1789760739.393694:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21127168 -> 21131264 00010000:00010000:0.0:1789760739.393710:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21127168->21131263] (req 21127168->21131263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.393724:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21127168->21131263] (req 21127168->21131263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.393733:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21127168->21131263] (req 21127168->21131263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.393741:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760739.393747:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21127168->21131263] (req 21127168->21131263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.393756:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21127168->21131263] (req 21127168->21131263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.393786:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21127168->21131263] (req 21127168->21131263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.393798:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086c7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21127168->21131263] (req 21127168->21131263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760739.393903:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484225024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35540us (35783us total) trans 4294972476 rc 0/0 00000100:00100000:0.0:1789760739.393914:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5187 00000100:00100000:0.0:1789760739.393923:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728659927397 time 728689277370 state 0x2 ref 0 00000100:00100000:0.0:1789760739.397594:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760739.397908:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484225152 00000100:00100000:1.0:1789760739.397918:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484225152 00000100:00100000:1.0:1789760739.398386:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728659927397 time 728689277370 state 0x0 ref 1 00000100:00100000:1.0:1789760739.398408:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728689537786 time 728689277370 state 0x1 ref 1 00000100:00100000:1.0:1789760739.398569:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728689537786 time 728729498730 state 0x0 ref 1 00000100:00100000:1.0:1789760739.398585:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484225152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760739.398885:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21131264, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:1.0:1789760739.398894:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.398909:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:1.0:1789760739.398913:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760739.398920:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21131264->21135359] (req 21131264->21135359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.398932:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21131264->21135359] (req 21131264->21135359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760739.431560:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972477 is committed 00002000:00010000:3.0:1789760739.431690:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21131264 -> 21135360 00010000:00010000:3.0:1789760739.431705:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21131264->21135359] (req 21131264->21135359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.431717:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21131264->21135359] (req 21131264->21135359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.431727:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21131264->21135359] (req 21131264->21135359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.431735:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760739.431739:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21131264->21135359] (req 21131264->21135359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.431749:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21131264->21135359] (req 21131264->21135359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.431769:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21131264->21135359] (req 21131264->21135359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.431777:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf38263962086ce lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21131264->21135359] (req 21131264->21135359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760739.431865:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484225152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33279us (34279us total) trans 4294972477 rc 0/0 00000100:00100000:3.0:1789760739.431876:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5188 00000100:00100000:3.0:1789760739.431885:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728689537786 time 728729498730 state 0x2 ref 0 00000100:00100000:2.0:1789760739.436021:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760739.436299:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484225280 00000100:00100000:1.0:1789760739.436309:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484225280 00000100:00100000:1.0:1789760739.436499:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728689537786 time 728729498730 state 0x0 ref 1 00000100:00100000:1.0:1789760739.436515:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728729759146 time 728729498730 state 0x1 ref 1 00000100:00100000:1.0:1789760739.436583:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728729759146 time 728767513142 state 0x0 ref 1 00000100:00100000:1.0:1789760739.436597:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484225280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760739.436643:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21135360, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:1.0:1789760739.436653:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.436669:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:1.0:1789760739.436674:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760739.436681:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21135360->21139455] (req 21135360->21139455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.436695:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21135360->21139455] (req 21135360->21139455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760739.477790:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972478 is committed 00002000:00010000:1.0:1789760739.478171:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21135360 -> 21139456 00010000:00010000:1.0:1789760739.478186:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21135360->21139455] (req 21135360->21139455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.478198:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21135360->21139455] (req 21135360->21139455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.478207:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21135360->21139455] (req 21135360->21139455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.478213:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760739.478218:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21135360->21139455] (req 21135360->21139455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.478226:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21135360->21139455] (req 21135360->21139455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.478245:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21135360->21139455] (req 21135360->21139455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.478253:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086d5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21135360->21139455] (req 21135360->21139455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760739.478347:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484225280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41748us (42330us total) trans 4294972478 rc 0/0 00000100:00100000:1.0:1789760739.478357:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5189 00000100:00100000:1.0:1789760739.478366:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728729759146 time 728767513142 state 0x2 ref 0 00000100:00100000:0.0:1789760739.484406:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760739.484617:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484225408 00000100:00100000:2.0:1789760739.484627:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484225408 00000100:00100000:2.0:1789760739.484676:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728729759146 time 728767513142 state 0x0 ref 1 00000100:00100000:2.0:1789760739.484690:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728767773558 time 728767513142 state 0x1 ref 1 00000100:00100000:2.0:1789760739.484719:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728767773558 time 728815650523 state 0x0 ref 1 00000100:00100000:2.0:1789760739.484731:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484225408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760739.484774:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21139456, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:2.0:1789760739.484782:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.484797:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:2.0:1789760739.484802:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760739.484808:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21139456->21143551] (req 21139456->21143551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.484821:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21139456->21143551] (req 21139456->21143551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760739.536978:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21139456 -> 21143552 00010000:00010000:2.0:1789760739.536998:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21139456->21143551] (req 21139456->21143551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.537011:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21139456->21143551] (req 21139456->21143551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.537020:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21139456->21143551] (req 21139456->21143551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.537027:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760739.537032:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21139456->21143551] (req 21139456->21143551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.537042:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21139456->21143551] (req 21139456->21143551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.537099:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21139456->21143551] (req 21139456->21143551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.537107:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086dc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21139456->21143551] (req 21139456->21143551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760739.537345:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484225408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52611us (52943us total) trans 4294972479 rc 0/0 00000100:00100000:2.0:1789760739.537355:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5190 00000100:00100000:2.0:1789760739.537365:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728767773558 time 728815650523 state 0x2 ref 0 00000001:00080000:0.0:1789760739.542351:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972479 is committed 00000100:00100000:2.0:1789760739.545895:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.545987:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484225536 00000100:00100000:3.0:1789760739.545997:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484225536 00000100:00100000:3.0:1789760739.546050:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728767773558 time 728815650523 state 0x0 ref 1 00000100:00100000:3.0:1789760739.546359:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728815910939 time 728815650523 state 0x1 ref 1 00000100:00100000:3.0:1789760739.546435:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728815910939 time 728877364988 state 0x0 ref 1 00000100:00100000:3.0:1789760739.546448:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484225536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.546496:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21143552, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760739.546504:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.546520:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:3.0:1789760739.546525:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.546530:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21143552->21147647] (req 21143552->21147647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.546543:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21143552->21147647] (req 21143552->21147647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760739.612194:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972480 is committed 00002000:00010000:0.0:1789760739.613301:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21143552 -> 21147648 00010000:00010000:0.0:1789760739.613318:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21143552->21147647] (req 21143552->21147647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.613342:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21143552->21147647] (req 21143552->21147647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.613351:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21143552->21147647] (req 21143552->21147647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.613358:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760739.613363:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21143552->21147647] (req 21143552->21147647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.613372:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21143552->21147647] (req 21143552->21147647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.613397:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21143552->21147647] (req 21143552->21147647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.613405:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962086e3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21143552->21147647] (req 21143552->21147647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760739.613568:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484225536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67118us (67678us total) trans 4294972480 rc 0/0 00000100:00100000:0.0:1789760739.613579:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5191 00000100:00100000:0.0:1789760739.613588:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728815910939 time 728877364988 state 0x2 ref 0 00000100:00100000:1.0:1789760739.620532:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.620614:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484225664 00000100:00100000:3.0:1789760739.620624:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484225664 00000100:00100000:3.0:1789760739.620679:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728815910939 time 728877364988 state 0x0 ref 1 00000100:00100000:3.0:1789760739.620694:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728877625404 time 728877364988 state 0x1 ref 1 00000100:00100000:3.0:1789760739.620722:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728877625404 time 728951653427 state 0x0 ref 1 00000100:00100000:3.0:1789760739.620735:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484225664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.620784:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21147648, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760739.620792:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.620807:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:3.0:1789760739.620811:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.620817:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21147648->21151743] (req 21147648->21151743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.620834:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21147648->21151743] (req 21147648->21151743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760739.658023:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21147648 -> 21151744 00010000:00010000:1.0:1789760739.658042:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21147648->21151743] (req 21147648->21151743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.658053:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21147648->21151743] (req 21147648->21151743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.658060:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21147648->21151743] (req 21147648->21151743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.658094:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760739.658099:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21147648->21151743] (req 21147648->21151743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.658108:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21147648->21151743] (req 21147648->21151743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.658137:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21147648->21151743] (req 21147648->21151743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.658144:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962086ea lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21147648->21151743] (req 21147648->21151743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760739.658232:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484225664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37494us (37706us total) trans 4294972481 rc 0/0 00000100:00100000:1.0:1789760739.658242:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5192 00000100:00100000:1.0:1789760739.658252:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728877625404 time 728951653427 state 0x2 ref 0 00000001:00080000:1.0:1789760739.658291:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972481 is committed 00000100:00100000:2.0:1789760739.661172:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760739.661288:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484225792 00000100:00100000:1.0:1789760739.661297:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484225792 00000100:00100000:1.0:1789760739.661362:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728877625404 time 728951653427 state 0x0 ref 1 00000100:00100000:1.0:1789760739.661377:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728951913843 time 728951653427 state 0x1 ref 1 00000100:00100000:1.0:1789760739.661406:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728951913843 time 728992337989 state 0x0 ref 1 00000100:00100000:1.0:1789760739.661426:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484225792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760739.661473:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21151744, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:1.0:1789760739.661482:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.661497:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=728s v=(0 0) 00010000:00010000:1.0:1789760739.661500:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760739.661506:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21151744->21155839] (req 21151744->21155839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.661519:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21151744->21155839] (req 21151744->21155839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760739.687522:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972482 is committed 00002000:00010000:3.0:1789760739.687590:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21151744 -> 21155840 00010000:00010000:3.0:1789760739.687606:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21151744->21155839] (req 21151744->21155839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.687620:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21151744->21155839] (req 21151744->21155839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.687630:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21151744->21155839] (req 21151744->21155839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.687638:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760739.687643:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21151744->21155839] (req 21151744->21155839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.687653:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21151744->21155839] (req 21151744->21155839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.687679:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21151744->21155839] (req 21151744->21155839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.687687:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962086f1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21151744->21155839] (req 21151744->21155839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760739.687823:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484225792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26402us (26667us total) trans 4294972482 rc 0/0 00000100:00100000:3.0:1789760739.687834:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5193 00000100:00100000:3.0:1789760739.687843:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728951913843 time 728992337989 state 0x2 ref 0 00000100:00100000:3.0:1789760739.690979:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760739.691198:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484225920 00000100:00100000:2.0:1789760739.691207:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484225920 00000100:00100000:2.0:1789760739.691260:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728951913843 time 728992337989 state 0x0 ref 1 00000100:00100000:2.0:1789760739.691275:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728992598405 time 728992337989 state 0x1 ref 1 00000100:00100000:2.0:1789760739.691356:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728992598405 time 729022286221 state 0x0 ref 1 00000100:00100000:2.0:1789760739.691368:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484225920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760739.691412:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21155840, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:2.0:1789760739.691421:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.691438:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:2.0:1789760739.691442:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760739.691448:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21155840->21159935] (req 21155840->21159935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.691462:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21155840->21159935] (req 21155840->21159935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760739.718976:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972483 is committed 00002000:00010000:2.0:1789760739.719106:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21155840 -> 21159936 00010000:00010000:2.0:1789760739.719131:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21155840->21159935] (req 21155840->21159935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.719144:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21155840->21159935] (req 21155840->21159935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.719158:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21155840->21159935] (req 21155840->21159935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.719165:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760739.719170:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21155840->21159935] (req 21155840->21159935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.719179:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21155840->21159935] (req 21155840->21159935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.719197:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21155840->21159935] (req 21155840->21159935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.719205:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962086f8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21155840->21159935] (req 21155840->21159935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760739.719285:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484225920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27914us (28311us total) trans 4294972483 rc 0/0 00000100:00100000:2.0:1789760739.719295:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5194 00000100:00100000:2.0:1789760739.719304:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728992598405 time 729022286221 state 0x2 ref 0 00000100:00100000:1.0:1789760739.722397:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760739.722500:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484226048 00000100:00100000:0.0:1789760739.722511:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484226048 00000100:00100000:0.0:1789760739.722568:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 728992598405 time 729022286221 state 0x0 ref 1 00000100:00100000:0.0:1789760739.722583:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729022546637 time 729022286221 state 0x1 ref 1 00000100:00100000:0.0:1789760739.722612:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729022546637 time 729053542562 state 0x0 ref 1 00000100:00100000:0.0:1789760739.722624:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760739.722668:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21159936, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:0.0:1789760739.722677:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.722698:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:0.0:1789760739.722703:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760739.722709:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21159936->21164031] (req 21159936->21164031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.722722:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21159936->21164031] (req 21159936->21164031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760739.750241:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972484 is committed 00002000:00010000:2.0:1789760739.750361:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21159936 -> 21164032 00010000:00010000:2.0:1789760739.750376:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21159936->21164031] (req 21159936->21164031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.750393:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21159936->21164031] (req 21159936->21164031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.750401:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21159936->21164031] (req 21159936->21164031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.750408:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760739.750413:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21159936->21164031] (req 21159936->21164031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.750421:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21159936->21164031] (req 21159936->21164031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.750442:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21159936->21164031] (req 21159936->21164031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.750449:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962086ff lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21159936->21164031] (req 21159936->21164031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760739.750545:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27919us (28151us total) trans 4294972484 rc 0/0 00000100:00100000:2.0:1789760739.750556:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5195 00000100:00100000:2.0:1789760739.750565:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729022546637 time 729053542562 state 0x2 ref 0 00000100:00100000:0.0:1789760739.754610:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760739.754749:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484226176 00000100:00100000:1.0:1789760739.754758:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484226176 00000100:00100000:1.0:1789760739.754811:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729022546637 time 729053542562 state 0x0 ref 1 00000100:00100000:1.0:1789760739.754826:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729053802978 time 729053542562 state 0x1 ref 1 00000100:00100000:1.0:1789760739.754887:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729053802978 time 729085817100 state 0x0 ref 1 00000100:00100000:1.0:1789760739.754900:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484226176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760739.754946:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21164032, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:1.0:1789760739.754954:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.754970:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:1.0:1789760739.754974:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760739.754981:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21164032->21168127] (req 21164032->21168127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.754996:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21164032->21168127] (req 21164032->21168127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760739.782140:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972485 is committed 00002000:00010000:1.0:1789760739.782290:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21164032 -> 21168128 00010000:00010000:1.0:1789760739.782306:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21164032->21168127] (req 21164032->21168127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.782320:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21164032->21168127] (req 21164032->21168127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.782340:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21164032->21168127] (req 21164032->21168127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.782348:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760739.782353:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21164032->21168127] (req 21164032->21168127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.782362:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21164032->21168127] (req 21164032->21168127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.782382:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21164032->21168127] (req 21164032->21168127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.782390:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208706 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21164032->21168127] (req 21164032->21168127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760739.782491:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484226176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27589us (27884us total) trans 4294972485 rc 0/0 00000100:00100000:1.0:1789760739.782502:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5196 00000100:00100000:1.0:1789760739.782511:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729053802978 time 729085817100 state 0x2 ref 0 00000100:00100000:0.0:1789760739.786165:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.786277:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484226304 00000100:00100000:3.0:1789760739.786286:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484226304 00000100:00100000:3.0:1789760739.786342:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729053802978 time 729085817100 state 0x0 ref 1 00000100:00100000:3.0:1789760739.786358:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729086077516 time 729085817100 state 0x1 ref 1 00000100:00100000:3.0:1789760739.786392:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729086077516 time 729117322970 state 0x0 ref 1 00000100:00100000:3.0:1789760739.786411:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.786519:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21168128, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760739.786527:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.786540:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:3.0:1789760739.786543:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.786547:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21168128->21172223] (req 21168128->21172223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.786558:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21168128->21172223] (req 21168128->21172223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760739.814168:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972486 is committed 00002000:00010000:3.0:1789760739.814232:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21168128 -> 21172224 00010000:00010000:3.0:1789760739.814246:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21168128->21172223] (req 21168128->21172223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.814259:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21168128->21172223] (req 21168128->21172223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.814269:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21168128->21172223] (req 21168128->21172223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.814276:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760739.814281:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21168128->21172223] (req 21168128->21172223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.814291:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21168128->21172223] (req 21168128->21172223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.814310:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21168128->21172223] (req 21168128->21172223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.814318:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620870d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21168128->21172223] (req 21168128->21172223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760739.814410:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27997us (28305us total) trans 4294972486 rc 0/0 00000100:00100000:3.0:1789760739.814491:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5197 00000100:00100000:3.0:1789760739.814501:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729086077516 time 729117322970 state 0x2 ref 0 00000100:00100000:2.0:1789760739.817532:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760739.817621:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484226432 00000100:00100000:0.0:1789760739.817630:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484226432 00000100:00100000:0.0:1789760739.817686:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729086077516 time 729117322970 state 0x0 ref 1 00000100:00100000:0.0:1789760739.817703:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729117583386 time 729117322970 state 0x1 ref 1 00000100:00100000:0.0:1789760739.817777:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729117583386 time 729148706734 state 0x0 ref 1 00000100:00100000:0.0:1789760739.817791:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484226432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760739.817838:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21172224, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:0.0:1789760739.817848:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.817863:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:0.0:1789760739.817868:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760739.817874:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21172224->21176319] (req 21172224->21176319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.817888:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21172224->21176319] (req 21172224->21176319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760739.851493:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972487 is committed 00002000:00010000:3.0:1789760739.851716:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21172224 -> 21176320 00010000:00010000:3.0:1789760739.851732:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21172224->21176319] (req 21172224->21176319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.851745:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21172224->21176319] (req 21172224->21176319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.851754:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21172224->21176319] (req 21172224->21176319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.851762:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760739.851766:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21172224->21176319] (req 21172224->21176319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.851775:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21172224->21176319] (req 21172224->21176319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.851793:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21172224->21176319] (req 21172224->21176319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.851801:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208714 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21172224->21176319] (req 21172224->21176319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760739.851914:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484226432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34121us (34384us total) trans 4294972487 rc 0/0 00000100:00100000:3.0:1789760739.851924:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5198 00000100:00100000:3.0:1789760739.851934:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729117583386 time 729148706734 state 0x2 ref 0 00000100:00100000:0.0:1789760739.855447:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760739.855850:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484226560 00000100:00100000:2.0:1789760739.855860:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484226560 00000100:00100000:2.0:1789760739.855913:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729117583386 time 729148706734 state 0x0 ref 1 00000100:00100000:2.0:1789760739.855929:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729148967150 time 729148706734 state 0x1 ref 1 00000100:00100000:2.0:1789760739.855958:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729148967150 time 729186888778 state 0x0 ref 1 00000100:00100000:2.0:1789760739.855970:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760739.856017:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21176320, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:2.0:1789760739.856026:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.856042:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:2.0:1789760739.856046:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760739.856051:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21176320->21180415] (req 21176320->21180415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.856100:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21176320->21180415] (req 21176320->21180415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760739.895002:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972488 is committed 00002000:00010000:2.0:1789760739.895363:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21176320 -> 21180416 00010000:00010000:2.0:1789760739.895379:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21176320->21180415] (req 21176320->21180415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.895391:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21176320->21180415] (req 21176320->21180415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.895400:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21176320->21180415] (req 21176320->21180415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.895407:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760739.895411:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21176320->21180415] (req 21176320->21180415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.895421:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21176320->21180415] (req 21176320->21180415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.895441:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21176320->21180415] (req 21176320->21180415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.895448:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620871b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21176320->21180415] (req 21176320->21180415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760739.895575:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39604us (40133us total) trans 4294972488 rc 0/0 00000100:00100000:2.0:1789760739.895586:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5199 00000100:00100000:2.0:1789760739.895595:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729148967150 time 729186888778 state 0x2 ref 0 00000100:00100000:0.0:1789760739.898952:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760739.899030:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484226688 00000100:00100000:1.0:1789760739.899040:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484226688 00000100:00100000:1.0:1789760739.899210:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729148967150 time 729186888778 state 0x0 ref 1 00000100:00100000:1.0:1789760739.899225:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729187149194 time 729186888778 state 0x1 ref 1 00000100:00100000:1.0:1789760739.899388:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729187149194 time 729230316783 state 0x0 ref 1 00000100:00100000:1.0:1789760739.899402:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760739.899456:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21180416, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:1.0:1789760739.899466:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.899481:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:1.0:1789760739.899486:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760739.899492:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21180416->21184511] (req 21180416->21184511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.899509:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21180416->21184511] (req 21180416->21184511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760739.928937:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972489 is committed 00002000:00010000:0.0:1789760739.929043:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21180416 -> 21184512 00010000:00010000:0.0:1789760739.929059:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21180416->21184511] (req 21180416->21184511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.929091:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21180416->21184511] (req 21180416->21184511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.929102:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21180416->21184511] (req 21180416->21184511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.929108:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760739.929124:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21180416->21184511] (req 21180416->21184511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.929133:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21180416->21184511] (req 21180416->21184511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.929153:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21180416->21184511] (req 21180416->21184511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760739.929162:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208722 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21180416->21184511] (req 21180416->21184511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760739.929255:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29851us (30307us total) trans 4294972489 rc 0/0 00000100:00100000:0.0:1789760739.929265:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5200 00000100:00100000:0.0:1789760739.929275:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729187149194 time 729230316783 state 0x2 ref 0 00000100:00100000:1.0:1789760739.933033:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.933242:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484226816 00000100:00100000:3.0:1789760739.933251:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484226816 00000100:00100000:3.0:1789760739.933302:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729187149194 time 729230316783 state 0x0 ref 1 00000100:00100000:3.0:1789760739.933317:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729230577199 time 729230316783 state 0x1 ref 1 00000100:00100000:3.0:1789760739.933396:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729230577199 time 729264326169 state 0x0 ref 1 00000100:00100000:3.0:1789760739.933413:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484226816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.933473:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21184512, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760739.933484:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.933500:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:3.0:1789760739.933504:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.933510:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21184512->21188607] (req 21184512->21188607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.933523:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21184512->21188607] (req 21184512->21188607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760739.957202:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21184512 -> 21188608 00010000:00010000:3.0:1789760739.957224:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21184512->21188607] (req 21184512->21188607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.957237:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21184512->21188607] (req 21184512->21188607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.957245:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21184512->21188607] (req 21184512->21188607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.957251:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760739.957255:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21184512->21188607] (req 21184512->21188607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.957264:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21184512->21188607] (req 21184512->21188607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.957283:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21184512->21188607] (req 21184512->21188607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.957291:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208729 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21184512->21188607] (req 21184512->21188607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760739.957382:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484226816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23967us (24355us total) trans 4294972490 rc 0/0 00000100:00100000:3.0:1789760739.957395:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5201 00000100:00100000:3.0:1789760739.957404:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729230577199 time 729264326169 state 0x2 ref 0 00000001:00080000:3.0:1789760739.959295:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972490 is committed 00000100:00100000:1.0:1789760739.960853:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760739.960936:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484226944 00000100:00100000:3.0:1789760739.960945:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484226944 00000100:00100000:3.0:1789760739.960995:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729230577199 time 729264326169 state 0x0 ref 1 00000100:00100000:3.0:1789760739.961009:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729264586585 time 729264326169 state 0x1 ref 1 00000100:00100000:3.0:1789760739.961088:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729264586585 time 729291995366 state 0x0 ref 1 00000100:00100000:3.0:1789760739.961101:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760739.961155:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21188608, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760739.961163:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.961177:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:3.0:1789760739.961181:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760739.961186:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21188608->21192703] (req 21188608->21192703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760739.961198:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21188608->21192703] (req 21188608->21192703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760739.990777:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972491 is committed 00002000:00010000:1.0:1789760739.990878:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21188608 -> 21192704 00010000:00010000:1.0:1789760739.990894:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21188608->21192703] (req 21188608->21192703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.990908:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21188608->21192703] (req 21188608->21192703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.990926:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21188608->21192703] (req 21188608->21192703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.990934:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760739.990939:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21188608->21192703] (req 21188608->21192703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.990949:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21188608->21192703] (req 21188608->21192703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.990967:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21188608->21192703] (req 21188608->21192703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760739.990980:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208730 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21188608->21192703] (req 21188608->21192703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760739.991104:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484226944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29965us (30218us total) trans 4294972491 rc 0/0 00000100:00100000:1.0:1789760739.991125:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5202 00000100:00100000:1.0:1789760739.991135:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729264586585 time 729291995366 state 0x2 ref 0 00000100:00100000:0.0:1789760739.997537:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760739.997642:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484227072 00000100:00100000:2.0:1789760739.997652:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484227072 00000100:00100000:2.0:1789760739.997706:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729264586585 time 729291995366 state 0x0 ref 1 00000100:00100000:2.0:1789760739.997721:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729292255782 time 729291995366 state 0x1 ref 1 00000100:00100000:2.0:1789760739.997787:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729292255782 time 729328717156 state 0x0 ref 1 00000100:00100000:2.0:1789760739.997799:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484227072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760739.997844:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21192704, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:2.0:1789760739.997853:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.997869:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:2.0:1789760739.997874:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760739.997880:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21192704->21196799] (req 21192704->21196799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760739.997895:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21192704->21196799] (req 21192704->21196799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760740.026845:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972492 is committed 00002000:00010000:3.0:1789760740.026887:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21192704 -> 21196800 00010000:00010000:3.0:1789760740.026902:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21192704->21196799] (req 21192704->21196799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.026915:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21192704->21196799] (req 21192704->21196799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.026924:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21192704->21196799] (req 21192704->21196799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.026930:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760740.026935:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21192704->21196799] (req 21192704->21196799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.026943:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21192704->21196799] (req 21192704->21196799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.026963:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21192704->21196799] (req 21192704->21196799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.026971:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208737 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21192704->21196799] (req 21192704->21196799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760740.027108:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484227072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29306us (29577us total) trans 4294972492 rc 0/0 00000100:00100000:3.0:1789760740.027117:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5203 00000100:00100000:3.0:1789760740.027127:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729292255782 time 729328717156 state 0x2 ref 0 00000100:00100000:2.0:1789760740.032426:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760740.032668:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484227200 00000100:00100000:0.0:1789760740.032678:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484227200 00000100:00080000:1.0:1789760740.032739:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760740.032747:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729292255782 time 729328717156 state 0x0 ref 1 00000100:00100000:0.0:1789760740.032763:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729328977572 time 729328717156 state 0x1 ref 1 00000100:00100000:0.0:1789760740.032792:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729328977572 time 729363722584 state 0x0 ref 1 00000100:00100000:0.0:1789760740.032810:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484227200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760740.032858:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21196800, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:0.0:1789760740.032867:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.032883:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:0.0:1789760740.032886:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760740.032890:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21196800->21200895] (req 21196800->21200895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.032901:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21196800->21200895] (req 21196800->21200895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760740.059477:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972493 is committed 00002000:00010000:0.0:1789760740.059936:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21196800 -> 21200896 00010000:00010000:0.0:1789760740.059952:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21196800->21200895] (req 21196800->21200895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.059966:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21196800->21200895] (req 21196800->21200895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.059975:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21196800->21200895] (req 21196800->21200895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.059983:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760740.059988:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21196800->21200895] (req 21196800->21200895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.059996:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21196800->21200895] (req 21196800->21200895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.060014:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21196800->21200895] (req 21196800->21200895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.060021:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620873e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21196800->21200895] (req 21196800->21200895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760740.060261:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484227200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27453us (27843us total) trans 4294972493 rc 0/0 00000100:00100000:0.0:1789760740.060272:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5204 00000100:00100000:0.0:1789760740.060282:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729328977572 time 729363722584 state 0x2 ref 0 00000100:00100000:1.0:1789760740.065791:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760740.066153:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484227328 00000100:00100000:3.0:1789760740.066164:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484227328 00000100:00100000:3.0:1789760740.066221:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729328977572 time 729363722584 state 0x0 ref 1 00000100:00100000:3.0:1789760740.066237:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729363983000 time 729363722584 state 0x1 ref 1 00000100:00100000:3.0:1789760740.066267:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729363983000 time 729397197565 state 0x0 ref 1 00000100:00100000:3.0:1789760740.066280:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484227328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760740.066335:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21200896, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760740.066344:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.066361:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:3.0:1789760740.066365:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760740.066371:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21200896->21204991] (req 21200896->21204991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.066385:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21200896->21204991] (req 21200896->21204991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760740.105059:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21200896 -> 21204992 00010000:00010000:3.0:1789760740.105113:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21200896->21204991] (req 21200896->21204991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.105127:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21200896->21204991] (req 21200896->21204991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.105136:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21200896->21204991] (req 21200896->21204991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.105142:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760740.105147:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21200896->21204991] (req 21200896->21204991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.105155:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21200896->21204991] (req 21200896->21204991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.105173:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21200896->21204991] (req 21200896->21204991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.105181:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208745 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21200896->21204991] (req 21200896->21204991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760740.105261:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484227328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38979us (39476us total) trans 4294972494 rc 0/0 00000100:00100000:3.0:1789760740.105271:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5205 00000100:00100000:3.0:1789760740.105281:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729363983000 time 729397197565 state 0x2 ref 0 00000001:00080000:1.0:1789760740.105920:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972494 is committed 00000100:00100000:1.0:1789760740.110976:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760740.111203:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484227456 00000100:00100000:3.0:1789760740.111214:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484227456 00000100:00100000:3.0:1789760740.111270:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729363983000 time 729397197565 state 0x0 ref 1 00000100:00100000:3.0:1789760740.111284:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729397457981 time 729397197565 state 0x1 ref 1 00000100:00100000:3.0:1789760740.111358:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729397457981 time 729442288401 state 0x0 ref 1 00000100:00100000:3.0:1789760740.111371:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484227456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760740.111490:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21204992, mtime: 0x6aad94e3, atime: 0x0, ctime: 0x6aad94e3, blocks: 0xa807 00010000:00010000:3.0:1789760740.111499:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.111515:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:3.0:1789760740.111520:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760740.111526:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21204992->21209087] (req 21204992->21209087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.111540:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21204992->21209087] (req 21204992->21209087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760740.139094:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972495 is committed 00002000:00010000:0.0:1789760740.139999:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21204992 -> 21209088 00002000:00010000:0.0:1789760740.140006:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760739 -> 1789760740 00002000:00010000:0.0:1789760740.140008:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760739 -> 1789760740 00010000:00010000:0.0:1789760740.140021:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21204992->21209087] (req 21204992->21209087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.140033:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21204992->21209087] (req 21204992->21209087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.140042:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21204992->21209087] (req 21204992->21209087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.140049:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760740.140054:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21204992->21209087] (req 21204992->21209087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.140062:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21204992->21209087] (req 21204992->21209087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.140096:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21204992->21209087] (req 21204992->21209087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.140104:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620874c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21204992->21209087] (req 21204992->21209087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760740.140224:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484227456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28849us (29253us total) trans 4294972495 rc 0/0 00000100:00100000:0.0:1789760740.140234:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5206 00000100:00100000:0.0:1789760740.140244:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729397457981 time 729442288401 state 0x2 ref 0 00000100:00100000:0.0:1789760740.149932:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760740.150145:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484227584 00000100:00100000:3.0:1789760740.150154:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484227584 00000100:00100000:3.0:1789760740.150205:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729397457981 time 729442288401 state 0x0 ref 1 00000100:00100000:3.0:1789760740.150220:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729442548817 time 729442288401 state 0x1 ref 1 00000100:00100000:3.0:1789760740.150251:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729442548817 time 729481182410 state 0x0 ref 1 00000100:00100000:3.0:1789760740.150264:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484227584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760740.150308:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21209088, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:3.0:1789760740.150317:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.150344:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:3.0:1789760740.150349:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760740.150356:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21209088->21213183] (req 21209088->21213183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.150371:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21209088->21213183] (req 21209088->21213183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760740.183041:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21209088 -> 21213184 00010000:00010000:1.0:1789760740.183061:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21209088->21213183] (req 21209088->21213183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.183202:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21209088->21213183] (req 21209088->21213183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.183211:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21209088->21213183] (req 21209088->21213183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.183219:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760740.183224:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21209088->21213183] (req 21209088->21213183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.183234:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21209088->21213183] (req 21209088->21213183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.183254:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21209088->21213183] (req 21209088->21213183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.183262:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208753 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21209088->21213183] (req 21209088->21213183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760740.183435:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972496 is committed 00000100:00100000:1.0:1789760740.183547:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484227584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33279us (33620us total) trans 4294972496 rc 0/0 00000100:00100000:1.0:1789760740.183557:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5207 00000100:00100000:1.0:1789760740.183566:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729442548817 time 729481182410 state 0x2 ref 0 00000100:00100000:1.0:1789760740.186516:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760740.186606:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484227712 00000100:00100000:0.0:1789760740.186657:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484227712 00000100:00100000:0.0:1789760740.186714:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729442548817 time 729481182410 state 0x0 ref 1 00000100:00100000:0.0:1789760740.186729:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729481442826 time 729481182410 state 0x1 ref 1 00000100:00100000:0.0:1789760740.186840:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729481442826 time 729517769158 state 0x0 ref 1 00000100:00100000:0.0:1789760740.186854:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484227712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760740.186898:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21213184, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:0.0:1789760740.186906:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.186921:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:0.0:1789760740.186925:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760740.186931:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21213184->21217279] (req 21213184->21217279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.186945:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21213184->21217279] (req 21213184->21217279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760740.236972:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972497 is committed 00002000:00010000:2.0:1789760740.237039:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21213184 -> 21217280 00010000:00010000:2.0:1789760740.237053:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21213184->21217279] (req 21213184->21217279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.237081:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21213184->21217279] (req 21213184->21217279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.237095:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21213184->21217279] (req 21213184->21217279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.237106:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760740.237111:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21213184->21217279] (req 21213184->21217279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.237121:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21213184->21217279] (req 21213184->21217279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.237141:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21213184->21217279] (req 21213184->21217279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.237149:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620875a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21213184->21217279] (req 21213184->21217279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760740.237244:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484227712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50388us (50732us total) trans 4294972497 rc 0/0 00000100:00100000:2.0:1789760740.237254:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5208 00000100:00100000:2.0:1789760740.237263:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729481442826 time 729517769158 state 0x2 ref 0 00000100:00100000:1.0:1789760740.245934:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760740.246027:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484227840 00000100:00100000:3.0:1789760740.246037:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484227840 00000100:00100000:3.0:1789760740.246123:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729481442826 time 729517769158 state 0x0 ref 1 00000100:00100000:3.0:1789760740.246140:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729518029574 time 729517769158 state 0x1 ref 1 00000100:00100000:3.0:1789760740.246166:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729518029574 time 729577096990 state 0x0 ref 1 00000100:00100000:3.0:1789760740.246180:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484227840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760740.246230:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21217280, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:3.0:1789760740.246238:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.246255:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:3.0:1789760740.246259:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760740.246265:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21217280->21221375] (req 21217280->21221375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.246278:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21217280->21221375] (req 21217280->21221375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.246954:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760740.316403:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972498 is committed 00002000:00010000:2.0:1789760740.316525:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21217280 -> 21221376 00010000:00010000:2.0:1789760740.316540:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21217280->21221375] (req 21217280->21221375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.316557:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21217280->21221375] (req 21217280->21221375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.316565:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21217280->21221375] (req 21217280->21221375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.316572:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760740.316577:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21217280->21221375] (req 21217280->21221375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.316587:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21217280->21221375] (req 21217280->21221375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.316606:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21217280->21221375] (req 21217280->21221375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.316614:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208761 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21217280->21221375] (req 21217280->21221375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760740.316704:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484227840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 70524us (70775us total) trans 4294972498 rc 0/0 00000100:00100000:2.0:1789760740.316715:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5209 00000100:00100000:2.0:1789760740.316725:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729518029574 time 729577096990 state 0x2 ref 0 00000100:00100000:0.0:1789760740.320242:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760740.320348:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484227968 00000100:00100000:3.0:1789760740.320358:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484227968 00000100:00100000:3.0:1789760740.320412:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729518029574 time 729577096990 state 0x0 ref 1 00000100:00100000:3.0:1789760740.320428:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729577357406 time 729577096990 state 0x1 ref 1 00000100:00100000:3.0:1789760740.320458:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729577357406 time 729651389069 state 0x0 ref 1 00000100:00100000:3.0:1789760740.320472:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484227968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760740.320525:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21221376, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:3.0:1789760740.320535:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.320552:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:3.0:1789760740.320556:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760740.320564:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21221376->21225471] (req 21221376->21225471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.320578:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21221376->21225471] (req 21221376->21225471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760740.399823:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21221376 -> 21225472 00010000:00010000:0.0:1789760740.399852:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21221376->21225471] (req 21221376->21225471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.399869:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21221376->21225471] (req 21221376->21225471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.399880:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21221376->21225471] (req 21221376->21225471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.399888:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760740.399894:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21221376->21225471] (req 21221376->21225471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.399905:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21221376->21225471] (req 21221376->21225471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.399934:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21221376->21225471] (req 21221376->21225471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.399947:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208768 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21221376->21225471] (req 21221376->21225471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760740.400152:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484227968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79674us (79912us total) trans 4294972499 rc 0/0 00000100:00100000:0.0:1789760740.400165:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5210 00000100:00100000:0.0:1789760740.400178:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729577357406 time 729651389069 state 0x2 ref 0 00000001:00080000:0.0:1789760740.400253:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972499 is committed 00000100:00100000:0.0:1789760740.404503:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760740.404876:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484228096 00000100:00100000:2.0:1789760740.404893:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484228096 00000100:00100000:2.0:1789760740.404972:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729577357406 time 729651389069 state 0x0 ref 1 00000100:00100000:2.0:1789760740.404992:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729651649485 time 729651389069 state 0x1 ref 1 00000100:00100000:2.0:1789760740.405027:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729651649485 time 729735957974 state 0x0 ref 1 00000100:00100000:2.0:1789760740.405043:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484228096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760740.405163:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21225472, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:2.0:1789760740.405174:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.405195:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:2.0:1789760740.405200:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760740.405208:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21225472->21229567] (req 21225472->21229567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.405226:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21225472->21229567] (req 21225472->21229567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760740.447598:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972500 is committed 00002000:00010000:0.0:1789760740.448306:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21225472 -> 21229568 00010000:00010000:0.0:1789760740.448332:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21225472->21229567] (req 21225472->21229567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.448345:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21225472->21229567] (req 21225472->21229567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.448354:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21225472->21229567] (req 21225472->21229567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.448362:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760740.448367:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21225472->21229567] (req 21225472->21229567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.448380:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21225472->21229567] (req 21225472->21229567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.448397:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21225472->21229567] (req 21225472->21229567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.448402:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620876f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21225472->21229567] (req 21225472->21229567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760740.448581:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484228096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43537us (44088us total) trans 4294972500 rc 0/0 00000100:00100000:0.0:1789760740.448591:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5211 00000100:00100000:0.0:1789760740.448601:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729651649485 time 729735957974 state 0x2 ref 0 00000100:00100000:0.0:1789760740.453989:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760740.454418:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484228224 00000100:00100000:2.0:1789760740.454428:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484228224 00000100:00100000:2.0:1789760740.454485:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729651649485 time 729735957974 state 0x0 ref 1 00000100:00100000:2.0:1789760740.454502:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729736218390 time 729735957974 state 0x1 ref 1 00000100:00100000:2.0:1789760740.454639:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729736218390 time 729785569019 state 0x0 ref 1 00000100:00100000:2.0:1789760740.454656:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706f050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484228224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760740.454707:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21229568, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:2.0:1789760740.454716:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.454732:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:2.0:1789760740.454736:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760740.454742:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21229568->21233663] (req 21229568->21233663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.454757:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21229568->21233663] (req 21229568->21233663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760740.501273:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972501 is committed 00002000:00010000:0.0:1789760740.501477:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21229568 -> 21233664 00010000:00010000:0.0:1789760740.501492:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21229568->21233663] (req 21229568->21233663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.501504:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21229568->21233663] (req 21229568->21233663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.501512:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21229568->21233663] (req 21229568->21233663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.501524:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760740.501529:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21229568->21233663] (req 21229568->21233663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.501537:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21229568->21233663] (req 21229568->21233663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.501555:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21229568->21233663] (req 21229568->21233663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.501562:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208776 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21229568->21233663] (req 21229568->21233663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760740.501707:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706f050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484228224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47049us (47722us total) trans 4294972501 rc 0/0 00000100:00100000:0.0:1789760740.501717:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5212 00000100:00100000:0.0:1789760740.501727:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729736218390 time 729785569019 state 0x2 ref 0 00000100:00100000:2.0:1789760740.511038:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760740.511166:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484228352 00000100:00100000:3.0:1789760740.511178:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484228352 00000100:00100000:3.0:1789760740.511234:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729736218390 time 729785569019 state 0x0 ref 1 00000100:00100000:3.0:1789760740.511250:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729785829435 time 729785569019 state 0x1 ref 1 00000100:00100000:3.0:1789760740.511269:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729785829435 time 729842201410 state 0x0 ref 1 00000100:00100000:3.0:1789760740.511280:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484228352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760740.511336:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21233664, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:3.0:1789760740.511346:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.511362:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:3.0:1789760740.511366:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760740.511373:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21233664->21237759] (req 21233664->21237759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.511390:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21233664->21237759] (req 21233664->21237759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760740.562860:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21233664 -> 21237760 00010000:00010000:0.0:1789760740.562879:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21233664->21237759] (req 21233664->21237759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.562892:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21233664->21237759] (req 21233664->21237759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.562906:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21233664->21237759] (req 21233664->21237759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.562913:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760740.562917:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21233664->21237759] (req 21233664->21237759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.562926:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21233664->21237759] (req 21233664->21237759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.562943:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21233664->21237759] (req 21233664->21237759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.562951:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620877d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21233664->21237759] (req 21233664->21237759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760740.563035:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484228352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51751us (52002us total) trans 4294972502 rc 0/0 00000100:00100000:0.0:1789760740.563046:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5213 00000100:00100000:0.0:1789760740.563055:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729785829435 time 729842201410 state 0x2 ref 0 00000001:00080000:1.0:1789760740.563752:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972502 is committed 00000100:00100000:1.0:1789760740.566306:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760740.566505:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484228480 00000100:00100000:0.0:1789760740.566516:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484228480 00000100:00100000:0.0:1789760740.566581:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729785829435 time 729842201410 state 0x0 ref 1 00000100:00100000:0.0:1789760740.566597:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729842461826 time 729842201410 state 0x1 ref 1 00000100:00100000:0.0:1789760740.566632:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729842461826 time 729897562651 state 0x0 ref 1 00000100:00100000:0.0:1789760740.566645:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484228480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760740.566691:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21237760, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:0.0:1789760740.566700:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.566716:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:0.0:1789760740.566721:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760740.566727:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21237760->21241855] (req 21237760->21241855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.566741:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21237760->21241855] (req 21237760->21241855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760740.605141:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972503 is committed 00002000:00010000:0.0:1789760740.605357:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21237760 -> 21241856 00010000:00010000:0.0:1789760740.605374:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21237760->21241855] (req 21237760->21241855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.605397:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21237760->21241855] (req 21237760->21241855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.605407:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21237760->21241855] (req 21237760->21241855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.605537:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760740.605542:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21237760->21241855] (req 21237760->21241855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.605551:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21237760->21241855] (req 21237760->21241855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.605570:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21237760->21241855] (req 21237760->21241855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.605578:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208784 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21237760->21241855] (req 21237760->21241855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760740.605674:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484228480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39027us (39372us total) trans 4294972503 rc 0/0 00000100:00100000:0.0:1789760740.605684:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5214 00000100:00100000:0.0:1789760740.605694:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729842461826 time 729897562651 state 0x2 ref 0 00000100:00100000:2.0:1789760740.609860:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760740.609953:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484228608 00000100:00100000:0.0:1789760740.609962:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484228608 00000100:00100000:0.0:1789760740.610021:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729842461826 time 729897562651 state 0x0 ref 1 00000100:00100000:0.0:1789760740.610037:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729897823067 time 729897562651 state 0x1 ref 1 00000100:00100000:0.0:1789760740.610189:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729897823067 time 729941117752 state 0x0 ref 1 00000100:00100000:0.0:1789760740.610204:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484228608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760740.610254:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21241856, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:0.0:1789760740.610263:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.610279:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:0.0:1789760740.610283:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760740.610289:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21241856->21245951] (req 21241856->21245951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.610303:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21241856->21245951] (req 21241856->21245951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760740.643808:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972504 is committed 00002000:00010000:0.0:1789760740.643875:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21241856 -> 21245952 00010000:00010000:0.0:1789760740.643891:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21241856->21245951] (req 21241856->21245951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.643902:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21241856->21245951] (req 21241856->21245951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.643910:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21241856->21245951] (req 21241856->21245951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.643917:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760740.643922:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21241856->21245951] (req 21241856->21245951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.643931:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21241856->21245951] (req 21241856->21245951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.643951:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21241856->21245951] (req 21241856->21245951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.643958:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620878b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21241856->21245951] (req 21241856->21245951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760740.644042:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484228608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33837us (34186us total) trans 4294972504 rc 0/0 00000100:00100000:0.0:1789760740.644051:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5215 00000100:00100000:0.0:1789760740.644061:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729897823067 time 729941117752 state 0x2 ref 0 00000100:00100000:2.0:1789760740.648997:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760740.649112:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484228736 00000100:00100000:0.0:1789760740.649122:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484228736 00000100:00100000:0.0:1789760740.649180:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729897823067 time 729941117752 state 0x0 ref 1 00000100:00100000:0.0:1789760740.649196:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729941378168 time 729941117752 state 0x1 ref 1 00000100:00100000:0.0:1789760740.649222:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729941378168 time 729980152875 state 0x0 ref 1 00000100:00100000:0.0:1789760740.649237:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484228736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760740.649286:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21245952, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:0.0:1789760740.649295:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.649312:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=729s v=(0 0) 00010000:00010000:0.0:1789760740.649317:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760740.649334:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21245952->21250047] (req 21245952->21250047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.649348:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21245952->21250047] (req 21245952->21250047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760740.708143:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972505 is committed 00002000:00010000:0.0:1789760740.708552:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21245952 -> 21250048 00010000:00010000:0.0:1789760740.708566:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21245952->21250047] (req 21245952->21250047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.708578:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21245952->21250047] (req 21245952->21250047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.708600:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21245952->21250047] (req 21245952->21250047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.708607:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760740.708612:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21245952->21250047] (req 21245952->21250047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.708621:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21245952->21250047] (req 21245952->21250047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.708642:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21245952->21250047] (req 21245952->21250047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.708650:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208792 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21245952->21250047] (req 21245952->21250047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760740.708763:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484228736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59524us (59772us total) trans 4294972505 rc 0/0 00000100:00100000:0.0:1789760740.708774:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5216 00000100:00100000:0.0:1789760740.708784:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729941378168 time 729980152875 state 0x2 ref 0 00000100:00100000:0.0:1789760740.714725:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760740.714819:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484228864 00000100:00100000:3.0:1789760740.714829:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484228864 00000100:00100000:3.0:1789760740.714882:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729941378168 time 729980152875 state 0x0 ref 1 00000100:00100000:3.0:1789760740.714898:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729980413291 time 729980152875 state 0x1 ref 1 00000100:00100000:3.0:1789760740.714929:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729980413291 time 730045859608 state 0x0 ref 1 00000100:00100000:3.0:1789760740.714947:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484228864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760740.714990:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21250048, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:3.0:1789760740.714999:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.715014:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:3.0:1789760740.715019:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760740.715024:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21250048->21254143] (req 21250048->21254143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.715037:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21250048->21254143] (req 21250048->21254143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760740.764220:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972506 is committed 00002000:00010000:3.0:1789760740.764402:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21250048 -> 21254144 00010000:00010000:3.0:1789760740.764418:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21250048->21254143] (req 21250048->21254143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.764431:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21250048->21254143] (req 21250048->21254143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.764440:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21250048->21254143] (req 21250048->21254143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.764448:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760740.764452:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21250048->21254143] (req 21250048->21254143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.764462:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21250048->21254143] (req 21250048->21254143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.764483:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21250048->21254143] (req 21250048->21254143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.764490:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208799 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21250048->21254143] (req 21250048->21254143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760740.764576:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484228864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49628us (49856us total) trans 4294972506 rc 0/0 00000100:00100000:3.0:1789760740.764587:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5217 00000100:00100000:3.0:1789760740.764597:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729980413291 time 730045859608 state 0x2 ref 0 00000100:00100000:2.0:1789760740.768872:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760740.768977:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484228992 00000100:00100000:3.0:1789760740.768986:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484228992 00000100:00100000:3.0:1789760740.769038:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 729980413291 time 730045859608 state 0x0 ref 1 00000100:00100000:3.0:1789760740.769052:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730046120024 time 730045859608 state 0x1 ref 1 00000100:00100000:3.0:1789760740.769144:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730046120024 time 730100074272 state 0x0 ref 1 00000100:00100000:3.0:1789760740.769158:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484228992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760740.769204:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21254144, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:3.0:1789760740.769214:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.769231:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:3.0:1789760740.769235:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760740.769241:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21254144->21258239] (req 21254144->21258239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.769255:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21254144->21258239] (req 21254144->21258239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760740.797979:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21254144 -> 21258240 00010000:00010000:1.0:1789760740.797998:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21254144->21258239] (req 21254144->21258239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.798013:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21254144->21258239] (req 21254144->21258239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.798024:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21254144->21258239] (req 21254144->21258239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.798032:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760740.798037:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21254144->21258239] (req 21254144->21258239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.798047:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21254144->21258239] (req 21254144->21258239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.798085:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21254144->21258239] (req 21254144->21258239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.798094:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962087a0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21254144->21258239] (req 21254144->21258239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760740.798205:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484228992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29043us (29336us total) trans 4294972507 rc 0/0 00000100:00100000:1.0:1789760740.798214:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5218 00000100:00100000:1.0:1789760740.798234:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730046120024 time 730100074272 state 0x2 ref 0 00000001:00080000:2.0:1789760740.798692:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972507 is committed 00000100:00100000:2.0:1789760740.808751:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760740.808844:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484229120 00000100:00100000:0.0:1789760740.808855:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484229120 00000100:00100000:0.0:1789760740.808914:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730046120024 time 730100074272 state 0x0 ref 1 00000100:00100000:0.0:1789760740.808927:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730100334688 time 730100074272 state 0x1 ref 1 00000100:00100000:0.0:1789760740.808947:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730100334688 time 730139878164 state 0x0 ref 1 00000100:00100000:0.0:1789760740.808959:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484229120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760740.809006:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21258240, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:0.0:1789760740.809014:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.809029:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:0.0:1789760740.809033:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760740.809040:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21258240->21262335] (req 21258240->21262335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760740.809054:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21258240->21262335] (req 21258240->21262335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760740.845386:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972508 is committed 00002000:00010000:2.0:1789760740.845590:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21258240 -> 21262336 00010000:00010000:2.0:1789760740.845605:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21258240->21262335] (req 21258240->21262335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.845618:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21258240->21262335] (req 21258240->21262335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.845628:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21258240->21262335] (req 21258240->21262335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.845635:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760740.845640:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21258240->21262335] (req 21258240->21262335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.845649:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21258240->21262335] (req 21258240->21262335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.845667:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21258240->21262335] (req 21258240->21262335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.845674:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087a7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21258240->21262335] (req 21258240->21262335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760740.845767:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484229120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36807us (37022us total) trans 4294972508 rc 0/0 00000100:00100000:2.0:1789760740.845776:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5219 00000100:00100000:2.0:1789760740.845784:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730100334688 time 730139878164 state 0x2 ref 0 00000100:00100000:0.0:1789760740.849565:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760740.849651:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484229248 00000100:00100000:2.0:1789760740.849662:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484229248 00000100:00100000:2.0:1789760740.849720:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730100334688 time 730139878164 state 0x0 ref 1 00000100:00100000:2.0:1789760740.849735:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730140138580 time 730139878164 state 0x1 ref 1 00000100:00100000:2.0:1789760740.849808:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730140138580 time 730180738191 state 0x0 ref 1 00000100:00100000:2.0:1789760740.849821:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484229248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760740.850004:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21262336, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:2.0:1789760740.850013:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.850030:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:2.0:1789760740.850034:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760740.850041:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21262336->21266431] (req 21262336->21266431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.850054:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21262336->21266431] (req 21262336->21266431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760740.878866:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21262336 -> 21266432 00010000:00010000:2.0:1789760740.878887:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21262336->21266431] (req 21262336->21266431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.878901:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21262336->21266431] (req 21262336->21266431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.878911:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21262336->21266431] (req 21262336->21266431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.878919:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760740.878925:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21262336->21266431] (req 21262336->21266431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.878934:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21262336->21266431] (req 21262336->21266431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.878953:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21262336->21266431] (req 21262336->21266431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.878962:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962087ae lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21262336->21266431] (req 21262336->21266431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760740.879046:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484229248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29222us (30080us total) trans 4294972509 rc 0/0 00000100:00100000:2.0:1789760740.879057:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5220 00000100:00100000:2.0:1789760740.879099:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730140138580 time 730180738191 state 0x2 ref 0 00000001:00080000:2.0:1789760740.879246:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972509 is committed 00000100:00100000:2.0:1789760740.882794:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760740.882914:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484229376 00000100:00100000:3.0:1789760740.882925:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484229376 00000100:00100000:3.0:1789760740.882982:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730140138580 time 730180738191 state 0x0 ref 1 00000100:00100000:3.0:1789760740.882998:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730180998607 time 730180738191 state 0x1 ref 1 00000100:00100000:3.0:1789760740.883060:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730180998607 time 730213990937 state 0x0 ref 1 00000100:00100000:3.0:1789760740.883110:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484229376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760740.883158:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21266432, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:3.0:1789760740.883167:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.883184:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:3.0:1789760740.883188:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760740.883194:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21266432->21270527] (req 21266432->21270527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760740.883213:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21266432->21270527] (req 21266432->21270527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760740.909954:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21266432 -> 21270528 00010000:00010000:2.0:1789760740.909974:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21266432->21270527] (req 21266432->21270527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.909987:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21266432->21270527] (req 21266432->21270527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.909996:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21266432->21270527] (req 21266432->21270527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.910005:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760740.910010:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21266432->21270527] (req 21266432->21270527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.910019:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21266432->21270527] (req 21266432->21270527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.910039:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21266432->21270527] (req 21266432->21270527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.910047:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087b5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21266432->21270527] (req 21266432->21270527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760740.910232:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484229376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27119us (27442us total) trans 4294972510 rc 0/0 00000100:00100000:2.0:1789760740.910242:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5221 00000100:00100000:2.0:1789760740.910252:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730180998607 time 730213990937 state 0x2 ref 0 00000001:00080000:2.0:1789760740.910290:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972510 is committed 00000100:00100000:2.0:1789760740.912889:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760740.912990:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484229504 00000100:00100000:1.0:1789760740.913000:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484229504 00000100:00100000:1.0:1789760740.913051:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730180998607 time 730213990937 state 0x0 ref 1 00000100:00100000:1.0:1789760740.913183:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730214251353 time 730213990937 state 0x1 ref 1 00000100:00100000:1.0:1789760740.913279:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730214251353 time 730244209431 state 0x0 ref 1 00000100:00100000:1.0:1789760740.913294:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484229504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760740.913354:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21270528, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:1.0:1789760740.913363:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.913379:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:1.0:1789760740.913383:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760740.913389:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21270528->21274623] (req 21270528->21274623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.913401:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21270528->21274623] (req 21270528->21274623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760740.949329:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972511 is committed 00002000:00010000:1.0:1789760740.949525:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21270528 -> 21274624 00010000:00010000:1.0:1789760740.949541:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21270528->21274623] (req 21270528->21274623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.949554:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21270528->21274623] (req 21270528->21274623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.949564:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21270528->21274623] (req 21270528->21274623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.949572:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760740.949578:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21270528->21274623] (req 21270528->21274623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.949586:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21270528->21274623] (req 21270528->21274623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.949607:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21270528->21274623] (req 21270528->21274623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760740.949683:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087bc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21270528->21274623] (req 21270528->21274623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760740.949820:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484229504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36523us (36933us total) trans 4294972511 rc 0/0 00000100:00100000:1.0:1789760740.949831:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5222 00000100:00100000:1.0:1789760740.949841:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730214251353 time 730244209431 state 0x2 ref 0 00000100:00100000:0.0:1789760740.954832:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760740.954914:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484229632 00000100:00100000:2.0:1789760740.954924:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484229632 00000100:00100000:2.0:1789760740.954976:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730214251353 time 730244209431 state 0x0 ref 1 00000100:00100000:2.0:1789760740.954992:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730244469847 time 730244209431 state 0x1 ref 1 00000100:00100000:2.0:1789760740.955027:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730244469847 time 730285957413 state 0x0 ref 1 00000100:00100000:2.0:1789760740.955040:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484229632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760740.955213:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21274624, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:2.0:1789760740.955223:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.955238:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:2.0:1789760740.955242:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760740.955249:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21274624->21278719] (req 21274624->21278719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760740.955262:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21274624->21278719] (req 21274624->21278719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760741.003759:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972512 is committed 00002000:00010000:0.0:1789760741.003816:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21274624 -> 21278720 00010000:00010000:0.0:1789760741.003840:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21274624->21278719] (req 21274624->21278719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.003853:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21274624->21278719] (req 21274624->21278719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.003862:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21274624->21278719] (req 21274624->21278719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.003869:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.003874:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21274624->21278719] (req 21274624->21278719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.003883:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21274624->21278719] (req 21274624->21278719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.003902:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21274624->21278719] (req 21274624->21278719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.003910:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962087c3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21274624->21278719] (req 21274624->21278719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.004004:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484229632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48963us (49177us total) trans 4294972512 rc 0/0 00000100:00100000:0.0:1789760741.004014:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5223 00000100:00100000:0.0:1789760741.004024:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730244469847 time 730285957413 state 0x2 ref 0 00000100:00100000:2.0:1789760741.006525:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.006662:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484229760 00000100:00100000:0.0:1789760741.006671:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484229760 00000100:00100000:0.0:1789760741.006724:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730244469847 time 730285957413 state 0x0 ref 1 00000100:00100000:0.0:1789760741.006738:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730286217829 time 730285957413 state 0x1 ref 1 00000100:00100000:0.0:1789760741.006767:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730286217829 time 730337698436 state 0x0 ref 1 00000100:00100000:0.0:1789760741.006779:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484229760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760741.006822:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21278720, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:0.0:1789760741.006893:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.006909:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:0.0:1789760741.006913:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760741.006919:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21278720->21282815] (req 21278720->21282815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.006933:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21278720->21282815] (req 21278720->21282815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760741.031135:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21278720 -> 21282816 00010000:00010000:0.0:1789760741.031156:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21278720->21282815] (req 21278720->21282815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.031169:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21278720->21282815] (req 21278720->21282815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.031178:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21278720->21282815] (req 21278720->21282815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.031186:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.031191:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21278720->21282815] (req 21278720->21282815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.031200:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21278720->21282815] (req 21278720->21282815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.031219:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21278720->21282815] (req 21278720->21282815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.031224:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962087ca lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21278720->21282815] (req 21278720->21282815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.031302:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484229760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24521us (24778us total) trans 4294972513 rc 0/0 00000100:00100000:0.0:1789760741.031316:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5224 00000100:00100000:0.0:1789760741.031336:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730286217829 time 730337698436 state 0x2 ref 0 00000001:00080000:1.0:1789760741.031419:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972513 is committed 00000100:00100000:3.0:1789760741.036139:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.036189:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484229888 00000100:00100000:0.0:1789760741.036204:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484229888 00000100:00100000:0.0:1789760741.036280:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730286217829 time 730337698436 state 0x0 ref 1 00000100:00100000:0.0:1789760741.036296:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730337958852 time 730337698436 state 0x1 ref 1 00000100:00100000:0.0:1789760741.036342:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730337958852 time 730367272734 state 0x0 ref 1 00000100:00100000:0.0:1789760741.036356:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484229888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760741.036509:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21282816, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:0.0:1789760741.036527:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.036544:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:0.0:1789760741.036549:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760741.036556:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21282816->21286911] (req 21282816->21286911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.036569:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21282816->21286911] (req 21282816->21286911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760741.081300:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21282816 -> 21286912 00010000:00010000:3.0:1789760741.081330:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21282816->21286911] (req 21282816->21286911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.081343:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21282816->21286911] (req 21282816->21286911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.081352:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21282816->21286911] (req 21282816->21286911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.081360:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760741.081365:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21282816->21286911] (req 21282816->21286911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.081374:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21282816->21286911] (req 21282816->21286911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.081460:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21282816->21286911] (req 21282816->21286911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.081469:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962087d1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21282816->21286911] (req 21282816->21286911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760741.081571:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484229888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45213us (45449us total) trans 4294972514 rc 0/0 00000100:00100000:3.0:1789760741.081581:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5225 00000100:00100000:3.0:1789760741.081590:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730337958852 time 730367272734 state 0x2 ref 0 00000001:00080000:2.0:1789760741.082300:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972514 is committed 00000100:00100000:2.0:1789760741.086294:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760741.086541:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484230016 00000100:00100000:1.0:1789760741.086551:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484230016 00000100:00100000:1.0:1789760741.086603:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730337958852 time 730367272734 state 0x0 ref 1 00000100:00100000:1.0:1789760741.086617:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730367533150 time 730367272734 state 0x1 ref 1 00000100:00100000:1.0:1789760741.086647:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730367533150 time 730417577847 state 0x0 ref 1 00000100:00100000:1.0:1789760741.086660:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484230016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760741.086705:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21286912, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:1.0:1789760741.086815:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.086830:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:1.0:1789760741.086895:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760741.086901:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21286912->21291007] (req 21286912->21291007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.086914:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21286912->21291007] (req 21286912->21291007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760741.125574:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972515 is committed 00002000:00010000:1.0:1789760741.125666:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21286912 -> 21291008 00010000:00010000:1.0:1789760741.125681:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21286912->21291007] (req 21286912->21291007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.125695:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21286912->21291007] (req 21286912->21291007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.125704:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21286912->21291007] (req 21286912->21291007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.125783:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760741.125788:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21286912->21291007] (req 21286912->21291007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.125797:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21286912->21291007] (req 21286912->21291007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.125815:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21286912->21291007] (req 21286912->21291007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.125821:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962087d8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21286912->21291007] (req 21286912->21291007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760741.125950:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484230016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39288us (39660us total) trans 4294972515 rc 0/0 00000100:00100000:1.0:1789760741.125959:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5226 00000100:00100000:1.0:1789760741.125969:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730367533150 time 730417577847 state 0x2 ref 0 00000100:00100000:0.0:1789760741.134703:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760741.135362:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484230144 00000100:00100000:2.0:1789760741.135372:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484230144 00000100:00100000:2.0:1789760741.135539:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730367533150 time 730417577847 state 0x0 ref 1 00000100:00100000:2.0:1789760741.135555:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730417838263 time 730417577847 state 0x1 ref 1 00000100:00100000:2.0:1789760741.135623:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730417838263 time 730466553510 state 0x0 ref 1 00000100:00100000:2.0:1789760741.135639:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484230144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760741.135686:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21291008, mtime: 0x6aad94e4, atime: 0x0, ctime: 0x6aad94e4, blocks: 0xa807 00010000:00010000:2.0:1789760741.135694:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.136501:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:2.0:1789760741.136507:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760741.136514:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21291008->21295103] (req 21291008->21295103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.136527:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21291008->21295103] (req 21291008->21295103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760741.186026:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972516 is committed 00002000:00010000:1.0:1789760741.186118:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21291008 -> 21295104 00002000:00010000:1.0:1789760741.186124:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760740 -> 1789760741 00002000:00010000:1.0:1789760741.186127:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760740 -> 1789760741 00010000:00010000:1.0:1789760741.186141:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21291008->21295103] (req 21291008->21295103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.186154:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21291008->21295103] (req 21291008->21295103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.186165:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21291008->21295103] (req 21291008->21295103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.186173:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760741.186178:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21291008->21295103] (req 21291008->21295103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.186188:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21291008->21295103] (req 21291008->21295103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.186209:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21291008->21295103] (req 21291008->21295103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.186217:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962087df lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21291008->21295103] (req 21291008->21295103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760741.186326:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484230144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50676us (51619us total) trans 4294972516 rc 0/0 00000100:00100000:1.0:1789760741.186336:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5227 00000100:00100000:1.0:1789760741.186346:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730417838263 time 730466553510 state 0x2 ref 0 00000100:00100000:3.0:1789760741.189920:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.190093:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484230272 00000100:00100000:0.0:1789760741.190103:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484230272 00000100:00100000:0.0:1789760741.190167:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730417838263 time 730466553510 state 0x0 ref 1 00000100:00100000:0.0:1789760741.190183:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730466813926 time 730466553510 state 0x1 ref 1 00000100:00100000:0.0:1789760741.190214:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730466813926 time 730521145685 state 0x0 ref 1 00000100:00100000:0.0:1789760741.190358:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484230272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760741.190410:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21295104, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:0.0:1789760741.190420:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.190437:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:0.0:1789760741.190442:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760741.190449:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21295104->21299199] (req 21295104->21299199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.190464:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21295104->21299199] (req 21295104->21299199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760741.220572:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21295104 -> 21299200 00010000:00010000:2.0:1789760741.220593:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21295104->21299199] (req 21295104->21299199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.220608:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21295104->21299199] (req 21295104->21299199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.220617:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21295104->21299199] (req 21295104->21299199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.220625:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760741.220630:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21295104->21299199] (req 21295104->21299199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.220640:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21295104->21299199] (req 21295104->21299199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.220660:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21295104->21299199] (req 21295104->21299199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.220669:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962087e6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21295104->21299199] (req 21295104->21299199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760741.220764:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484230272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30535us (30848us total) trans 4294972517 rc 0/0 00000100:00100000:2.0:1789760741.220775:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5228 00000100:00100000:2.0:1789760741.220785:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730466813926 time 730521145685 state 0x2 ref 0 00000001:00080000:0.0:1789760741.221306:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972517 is committed 00000100:00100000:0.0:1789760741.223258:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760741.223392:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484230400 00000100:00100000:3.0:1789760741.223403:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484230400 00000100:00100000:3.0:1789760741.223456:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730466813926 time 730521145685 state 0x0 ref 1 00000100:00100000:3.0:1789760741.223471:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730521406101 time 730521145685 state 0x1 ref 1 00000100:00100000:3.0:1789760741.223505:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730521406101 time 730554435397 state 0x0 ref 1 00000100:00100000:3.0:1789760741.223517:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484230400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760741.223562:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21299200, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:3.0:1789760741.223571:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.223586:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:3.0:1789760741.223591:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760741.223597:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21299200->21303295] (req 21299200->21303295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.223616:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21299200->21303295] (req 21299200->21303295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760741.245094:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972518 is committed 00002000:00010000:3.0:1789760741.245312:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21299200 -> 21303296 00010000:00010000:3.0:1789760741.245339:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21299200->21303295] (req 21299200->21303295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.245352:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21299200->21303295] (req 21299200->21303295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.245361:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21299200->21303295] (req 21299200->21303295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.245369:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760741.245373:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21299200->21303295] (req 21299200->21303295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.245382:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21299200->21303295] (req 21299200->21303295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.245401:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21299200->21303295] (req 21299200->21303295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.245410:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962087ed lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21299200->21303295] (req 21299200->21303295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760741.245509:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484230400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 21989us (22254us total) trans 4294972518 rc 0/0 00000100:00100000:3.0:1789760741.245520:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5229 00000100:00100000:3.0:1789760741.245530:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730521406101 time 730554435397 state 0x2 ref 0 00000100:00100000:0.0:1789760741.250654:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760741.250737:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484230528 00000100:00100000:1.0:1789760741.250748:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484230528 00000100:00100000:1.0:1789760741.250802:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730521406101 time 730554435397 state 0x0 ref 1 00000100:00100000:1.0:1789760741.250817:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730554695813 time 730554435397 state 0x1 ref 1 00000100:00100000:1.0:1789760741.250896:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730554695813 time 730581826978 state 0x0 ref 1 00000100:00100000:1.0:1789760741.250908:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484230528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760741.250954:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21303296, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:1.0:1789760741.250964:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.250979:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:1.0:1789760741.250983:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760741.250989:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21303296->21307391] (req 21303296->21307391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.251002:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21303296->21307391] (req 21303296->21307391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.263614:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:1.0:1789760741.274452:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21303296 -> 21307392 00010000:00010000:1.0:1789760741.274472:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21303296->21307391] (req 21303296->21307391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.274486:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21303296->21307391] (req 21303296->21307391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.274495:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21303296->21307391] (req 21303296->21307391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.274501:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760741.274507:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21303296->21307391] (req 21303296->21307391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.274516:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21303296->21307391] (req 21303296->21307391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.274540:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21303296->21307391] (req 21303296->21307391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.274547:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962087f4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21303296->21307391] (req 21303296->21307391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760741.274644:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484230528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23731us (23994us total) trans 4294972519 rc 0/0 00000100:00100000:1.0:1789760741.274654:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5230 00000100:00100000:1.0:1789760741.274664:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730554695813 time 730581826978 state 0x2 ref 0 00000001:00080000:1.0:1789760741.274700:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972519 is committed 00000100:00100000:2.0:1789760741.277865:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760741.282515:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484230656 00000100:00100000:1.0:1789760741.282530:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484230656 00000100:00100000:1.0:1789760741.282593:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730554695813 time 730581826978 state 0x0 ref 1 00000100:00100000:1.0:1789760741.282608:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730582087394 time 730581826978 state 0x1 ref 1 00000100:00100000:1.0:1789760741.282641:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730582087394 time 730613571569 state 0x0 ref 1 00000100:00100000:1.0:1789760741.282655:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484230656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760741.282702:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21307392, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:1.0:1789760741.282711:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.282731:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:1.0:1789760741.282736:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760741.282742:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21307392->21311487] (req 21307392->21311487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.282755:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21307392->21311487] (req 21307392->21311487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760741.313221:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972520 is committed 00002000:00010000:3.0:1789760741.313390:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21307392 -> 21311488 00010000:00010000:3.0:1789760741.313406:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21307392->21311487] (req 21307392->21311487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.313419:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21307392->21311487] (req 21307392->21311487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.313428:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21307392->21311487] (req 21307392->21311487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.313437:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760741.313441:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21307392->21311487] (req 21307392->21311487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.313451:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21307392->21311487] (req 21307392->21311487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.313471:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21307392->21311487] (req 21307392->21311487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.313484:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962087fb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21307392->21311487] (req 21307392->21311487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760741.313579:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484230656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30923us (35718us total) trans 4294972520 rc 0/0 00000100:00100000:3.0:1789760741.313589:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5231 00000100:00100000:3.0:1789760741.313598:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730582087394 time 730613571569 state 0x2 ref 0 00000100:00100000:3.0:1789760741.316237:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760741.316328:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484230784 00000100:00100000:2.0:1789760741.316338:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484230784 00000100:00100000:2.0:1789760741.316389:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730582087394 time 730613571569 state 0x0 ref 1 00000100:00100000:2.0:1789760741.316403:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730613831985 time 730613571569 state 0x1 ref 1 00000100:00100000:2.0:1789760741.316481:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730613831985 time 730647410944 state 0x0 ref 1 00000100:00100000:2.0:1789760741.316494:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484230784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760741.316546:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21311488, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:2.0:1789760741.316554:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.316570:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:2.0:1789760741.316575:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760741.316581:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21311488->21315583] (req 21311488->21315583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.316595:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21311488->21315583] (req 21311488->21315583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760741.359809:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21311488 -> 21315584 00010000:00010000:0.0:1789760741.359830:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21311488->21315583] (req 21311488->21315583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.359843:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21311488->21315583] (req 21311488->21315583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.359853:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21311488->21315583] (req 21311488->21315583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.359860:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.359865:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21311488->21315583] (req 21311488->21315583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.359874:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21311488->21315583] (req 21311488->21315583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.359893:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21311488->21315583] (req 21311488->21315583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.359900:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208802 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21311488->21315583] (req 21311488->21315583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760741.360162:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972521 is committed 00000100:00100000:0.0:1789760741.360234:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484230784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43738us (44001us total) trans 4294972521 rc 0/0 00000100:00100000:0.0:1789760741.360244:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5232 00000100:00100000:0.0:1789760741.360254:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730613831985 time 730647410944 state 0x2 ref 0 00000100:00100000:0.0:1789760741.367354:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760741.368551:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484230912 00000100:00100000:3.0:1789760741.368570:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484230912 00000100:00100000:3.0:1789760741.368655:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730613831985 time 730647410944 state 0x0 ref 1 00000100:00100000:3.0:1789760741.368676:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730647671360 time 730647410944 state 0x1 ref 1 00000100:00100000:3.0:1789760741.368801:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730647671360 time 730699723457 state 0x0 ref 1 00000100:00100000:3.0:1789760741.368817:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484230912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760741.368886:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21315584, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:3.0:1789760741.368897:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.368918:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:3.0:1789760741.368923:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760741.368931:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21315584->21319679] (req 21315584->21319679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.368948:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21315584->21319679] (req 21315584->21319679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760741.418974:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972522 is committed 00002000:00010000:0.0:1789760741.419152:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21315584 -> 21319680 00010000:00010000:0.0:1789760741.419167:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21315584->21319679] (req 21315584->21319679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.419181:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21315584->21319679] (req 21315584->21319679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.419191:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21315584->21319679] (req 21315584->21319679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.419199:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.419204:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21315584->21319679] (req 21315584->21319679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.419214:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21315584->21319679] (req 21315584->21319679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.419234:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21315584->21319679] (req 21315584->21319679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.419260:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208809 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21315584->21319679] (req 21315584->21319679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.419358:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484230912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50540us (52012us total) trans 4294972522 rc 0/0 00000100:00100000:0.0:1789760741.419368:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5233 00000100:00100000:0.0:1789760741.419378:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730647671360 time 730699723457 state 0x2 ref 0 00000100:00100000:2.0:1789760741.422586:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760741.422858:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484231040 00000100:00100000:1.0:1789760741.422869:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484231040 00000100:00100000:1.0:1789760741.422924:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730647671360 time 730699723457 state 0x0 ref 1 00000100:00100000:1.0:1789760741.422940:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730699983873 time 730699723457 state 0x1 ref 1 00000100:00100000:1.0:1789760741.423010:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730699983873 time 730753940000 state 0x0 ref 1 00000100:00100000:1.0:1789760741.423025:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484231040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760741.423115:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21319680, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:1.0:1789760741.423124:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.423140:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:1.0:1789760741.423145:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760741.423151:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21319680->21323775] (req 21319680->21323775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.423165:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21319680->21323775] (req 21319680->21323775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760741.493534:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972523 is committed 00002000:00010000:0.0:1789760741.496236:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21319680 -> 21323776 00010000:00010000:0.0:1789760741.496256:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21319680->21323775] (req 21319680->21323775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.496270:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21319680->21323775] (req 21319680->21323775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.496279:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21319680->21323775] (req 21319680->21323775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.496287:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.496291:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21319680->21323775] (req 21319680->21323775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.496301:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21319680->21323775] (req 21319680->21323775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.496388:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21319680->21323775] (req 21319680->21323775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.496397:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208810 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21319680->21323775] (req 21319680->21323775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.496490:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484231040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73463us (73908us total) trans 4294972523 rc 0/0 00000100:00100000:0.0:1789760741.496502:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5234 00000100:00100000:0.0:1789760741.496513:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730699983873 time 730753940000 state 0x2 ref 0 00000100:00100000:2.0:1789760741.502042:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760741.502223:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484231168 00000100:00100000:1.0:1789760741.502233:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484231168 00000100:00100000:1.0:1789760741.502290:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730699983873 time 730753940000 state 0x0 ref 1 00000100:00100000:1.0:1789760741.502306:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730754200416 time 730753940000 state 0x1 ref 1 00000100:00100000:1.0:1789760741.502345:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730754200416 time 730833276178 state 0x0 ref 1 00000100:00100000:1.0:1789760741.502358:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484231168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760741.502404:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21323776, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:1.0:1789760741.502413:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.502427:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:1.0:1789760741.502431:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760741.502437:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21323776->21327871] (req 21323776->21327871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.502449:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21323776->21327871] (req 21323776->21327871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760741.563738:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21323776 -> 21327872 00010000:00010000:1.0:1789760741.563756:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21323776->21327871] (req 21323776->21327871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.563769:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21323776->21327871] (req 21323776->21327871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.563779:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21323776->21327871] (req 21323776->21327871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.563787:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760741.563792:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21323776->21327871] (req 21323776->21327871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.563803:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21323776->21327871] (req 21323776->21327871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.563823:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21323776->21327871] (req 21323776->21327871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.563831:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208817 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21323776->21327871] (req 21323776->21327871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760741.563936:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484231168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61576us (61899us total) trans 4294972524 rc 0/0 00000100:00100000:1.0:1789760741.563950:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5235 00000100:00100000:1.0:1789760741.563960:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730754200416 time 730833276178 state 0x2 ref 0 00000001:00080000:2.0:1789760741.564532:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972524 is committed 00000100:00100000:1.0:1789760741.567833:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.568484:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484231296 00000100:00100000:0.0:1789760741.568496:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484231296 00000100:00100000:2.0:1789760741.568637:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730754200416 time 730833276178 state 0x0 ref 1 00000100:00100000:2.0:1789760741.568652:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730833536594 time 730833276178 state 0x1 ref 1 00000100:00100000:2.0:1789760741.568720:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730833536594 time 730899650300 state 0x0 ref 1 00000100:00100000:2.0:1789760741.568733:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484231296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760741.568781:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21327872, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:2.0:1789760741.568789:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.568811:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:2.0:1789760741.568815:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760741.568821:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21327872->21331967] (req 21327872->21331967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.568834:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21327872->21331967] (req 21327872->21331967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760741.614180:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21327872 -> 21331968 00000001:00080000:1.0:1789760741.614188:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972525 is committed 00010000:00010000:0.0:1789760741.614201:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21327872->21331967] (req 21327872->21331967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.614396:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21327872->21331967] (req 21327872->21331967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.614407:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21327872->21331967] (req 21327872->21331967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.614413:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.614417:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21327872->21331967] (req 21327872->21331967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.614424:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21327872->21331967] (req 21327872->21331967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.614441:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21327872->21331967] (req 21327872->21331967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.614446:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620881e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21327872->21331967] (req 21327872->21331967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.614528:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484231296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45793us (46700us total) trans 4294972525 rc 0/0 00000100:00100000:0.0:1789760741.614539:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5236 00000100:00100000:0.0:1789760741.614549:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730833536594 time 730899650300 state 0x2 ref 0 00000100:00100000:0.0:1789760741.624017:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760741.624340:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484231424 00000100:00100000:2.0:1789760741.624350:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484231424 00000100:00100000:2.0:1789760741.624411:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730833536594 time 730899650300 state 0x0 ref 1 00000100:00100000:2.0:1789760741.624427:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730899910716 time 730899650300 state 0x1 ref 1 00000100:00100000:2.0:1789760741.624459:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730899910716 time 730955389822 state 0x0 ref 1 00000100:00100000:2.0:1789760741.624471:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484231424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760741.624517:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21331968, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:2.0:1789760741.624525:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.624540:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:2.0:1789760741.624544:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760741.624550:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21331968->21336063] (req 21331968->21336063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.624568:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21331968->21336063] (req 21331968->21336063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760741.656773:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972526 is committed 00002000:00010000:2.0:1789760741.656920:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21331968 -> 21336064 00010000:00010000:2.0:1789760741.656936:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21331968->21336063] (req 21331968->21336063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.656950:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21331968->21336063] (req 21331968->21336063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.656959:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21331968->21336063] (req 21331968->21336063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.656967:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760741.656972:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21331968->21336063] (req 21331968->21336063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.656982:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21331968->21336063] (req 21331968->21336063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.657000:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21331968->21336063] (req 21331968->21336063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.657006:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208825 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21331968->21336063] (req 21331968->21336063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760741.657279:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484231424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32805us (33266us total) trans 4294972526 rc 0/0 00000100:00100000:2.0:1789760741.657290:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5237 00000100:00100000:2.0:1789760741.657300:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730899910716 time 730955389822 state 0x2 ref 0 00000100:00100000:2.0:1789760741.660427:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.660504:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484231552 00000100:00100000:0.0:1789760741.660514:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484231552 00000100:00100000:0.0:1789760741.660573:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730899910716 time 730955389822 state 0x0 ref 1 00000100:00100000:0.0:1789760741.660588:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730955650238 time 730955389822 state 0x1 ref 1 00000100:00100000:0.0:1789760741.660621:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730955650238 time 730991552119 state 0x0 ref 1 00000100:00100000:0.0:1789760741.660634:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484231552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760741.660680:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21336064, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:0.0:1789760741.660688:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.660703:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=730s v=(0 0) 00010000:00010000:0.0:1789760741.660708:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760741.660714:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21336064->21340159] (req 21336064->21340159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.660728:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21336064->21340159] (req 21336064->21340159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760741.690775:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972527 is committed 00002000:00010000:1.0:1789760741.690862:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21336064 -> 21340160 00010000:00010000:1.0:1789760741.690877:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21336064->21340159] (req 21336064->21340159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.690889:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21336064->21340159] (req 21336064->21340159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.690898:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21336064->21340159] (req 21336064->21340159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.690905:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760741.690911:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21336064->21340159] (req 21336064->21340159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.690920:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21336064->21340159] (req 21336064->21340159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.690941:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21336064->21340159] (req 21336064->21340159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.690949:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620882c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21336064->21340159] (req 21336064->21340159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760741.691125:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484231552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30488us (30701us total) trans 4294972527 rc 0/0 00000100:00100000:1.0:1789760741.691136:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5238 00000100:00100000:1.0:1789760741.691145:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730955650238 time 730991552119 state 0x2 ref 0 00000100:00100000:1.0:1789760741.695358:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.695527:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484231680 00000100:00100000:0.0:1789760741.695537:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484231680 00000100:00100000:0.0:1789760741.695601:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730955650238 time 730991552119 state 0x0 ref 1 00000100:00100000:0.0:1789760741.695617:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730991812535 time 730991552119 state 0x1 ref 1 00000100:00100000:0.0:1789760741.695649:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730991812535 time 731026579654 state 0x0 ref 1 00000100:00100000:0.0:1789760741.695662:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484231680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760741.695714:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21340160, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:0.0:1789760741.695723:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.695741:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:0.0:1789760741.695809:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760741.695817:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21340160->21344255] (req 21340160->21344255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.695832:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21340160->21344255] (req 21340160->21344255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760741.736140:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21340160 -> 21344256 00010000:00010000:0.0:1789760741.736159:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21340160->21344255] (req 21340160->21344255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.736177:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21340160->21344255] (req 21340160->21344255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.736187:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21340160->21344255] (req 21340160->21344255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.736195:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.736200:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21340160->21344255] (req 21340160->21344255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.736209:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21340160->21344255] (req 21340160->21344255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.736229:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21340160->21344255] (req 21340160->21344255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.736238:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208833 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21340160->21344255] (req 21340160->21344255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.736340:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484231680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40676us (41056us total) trans 4294972528 rc 0/0 00000100:00100000:0.0:1789760741.736356:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5239 00000100:00100000:0.0:1789760741.736365:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730991812535 time 731026579654 state 0x2 ref 0 00000001:00080000:1.0:1789760741.736762:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972528 is committed 00000100:00100000:0.0:1789760741.740604:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760741.741521:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484231808 00000100:00100000:2.0:1789760741.741532:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484231808 00000100:00100000:2.0:1789760741.741586:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 730991812535 time 731026579654 state 0x0 ref 1 00000100:00100000:2.0:1789760741.741602:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731026840070 time 731026579654 state 0x1 ref 1 00000100:00100000:2.0:1789760741.741637:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731026840070 time 731072567852 state 0x0 ref 1 00000100:00100000:2.0:1789760741.741651:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484231808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760741.741699:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21344256, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:2.0:1789760741.741708:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.741724:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:2.0:1789760741.741728:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760741.741735:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21344256->21348351] (req 21344256->21348351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.741749:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21344256->21348351] (req 21344256->21348351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760741.768500:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972529 is committed 00002000:00010000:0.0:1789760741.768600:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21344256 -> 21348352 00010000:00010000:0.0:1789760741.768615:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21344256->21348351] (req 21344256->21348351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.768628:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21344256->21348351] (req 21344256->21348351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.768638:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21344256->21348351] (req 21344256->21348351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.768645:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.768650:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21344256->21348351] (req 21344256->21348351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.768660:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21344256->21348351] (req 21344256->21348351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.768680:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21344256->21348351] (req 21344256->21348351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.768687:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620883a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21344256->21348351] (req 21344256->21348351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.768782:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484231808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27128us (28183us total) trans 4294972529 rc 0/0 00000100:00100000:0.0:1789760741.768793:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5240 00000100:00100000:0.0:1789760741.768802:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731026840070 time 731072567852 state 0x2 ref 0 00000100:00100000:0.0:1789760741.772440:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508920192:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760741.772513:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.772736:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c45500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508920576:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.772768:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.772789:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c47480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508921088:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.772813:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.772831:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9472a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508921600:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.772851:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.772869:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1789760741.772875:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c47100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508920704:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.772894:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.772914:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c47800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508921216:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.772958:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.772981:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9470e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508921728:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.773004:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.773027:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c44e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3481:1876699508920448:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.773050:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.773095:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c45880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3481:1876699508920960:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.773116:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.773132:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9471880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3481:1876699508921472:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.773149:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.773166:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [2->3] 00000100:00100000:1.0:1789760741.773171:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c47b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508920320:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.773196:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.773216:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c45180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508920832:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.773235:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760741.773402:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508920192 00000100:00100000:2.0:1789760741.773410:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508920192 00000100:00100000:2.0:1789760741.773469:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 380 00000100:00100000:2.0:1789760741.773479:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508920192:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1789760741.773514:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760741.773560:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508920192:12345-0@lo:13:osp-pre-1-0.0 Request processed in 82us (1046us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760741.773567:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 380 00000100:00080000:0.0:1789760741.773624:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:3.0:1789760741.774622:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8910380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508921856:0@lo:13:osp-pre-1-1.0 00000100:00100000:3.0:1789760741.774683:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760741.774737:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508921856 00000100:00100000:3.0:1789760741.774742:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508921856 00000100:00100000:3.0:1789760741.774795:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 381 00000100:00100000:3.0:1789760741.774804:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508921856:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:3.0:1789760741.774820:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:3.0:1789760741.774876:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508921856:12345-0@lo:13:osp-pre-1-1.0 Request processed in 73us (194us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760741.774885:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 381 00000100:00080000:3.0:1789760741.774911:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:3.0:1789760741.774949:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8910380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508921856:0@lo:13:osp-pre-1-1.0 00000100:00100000:1.0:1789760741.776394:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c7c44700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508921344:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.776452:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760741.776507:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508920576 00000100:00100000:1.0:1789760741.776513:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508920576 00000100:00100000:1.0:1789760741.776557:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508920832 00000100:00100000:1.0:1789760741.776560:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508920832 00000100:00100000:1.0:1789760741.776576:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 439 00000100:00100000:1.0:1789760741.776583:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508920576:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.776635:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470380 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508920576:12345-0@lo:400:kworker.0 Request processed in 52us (3866us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760741.776642:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 439 00000100:00100000:1.0:1789760741.776657:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 440 00000100:00100000:1.0:1789760741.776663:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508920832:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.776694:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508920832:12345-0@lo:400:kworker.0 Request processed in 32us (3458us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760741.776700:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 440 00000100:00100000:1.0:1789760741.776719:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508921088 00000100:00100000:1.0:1789760741.776724:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508921088 00000100:00100000:1.0:1789760741.776754:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508921600 00000100:00100000:1.0:1789760741.776756:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508921600 00000100:00100000:1.0:1789760741.776774:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508920704 00000100:00100000:1.0:1789760741.776777:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508920704 00000100:00100000:1.0:1789760741.776786:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508920448 00000100:00100000:1.0:1789760741.776789:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508920448 00000100:00100000:1.0:1789760741.776852:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508920960 00000100:00100000:1.0:1789760741.776856:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508920960 00000100:00100000:1.0:1789760741.776868:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508921472 00000100:00100000:1.0:1789760741.776871:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508921472 00000100:00100000:1.0:1789760741.776883:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 915 00000100:00100000:1.0:1789760741.776890:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508921088:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.776923:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508921088:12345-0@lo:400:kworker.0 Request processed in 34us (4109us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760741.776930:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 915 00000100:00100000:1.0:1789760741.776941:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 916 00000100:00100000:1.0:1789760741.776947:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508921600:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.776973:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508921600:12345-0@lo:400:kworker.0 Request processed in 26us (4120us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760741.776978:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 916 00000100:00100000:1.0:1789760741.776989:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 917 00000100:00100000:1.0:1789760741.776995:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508920704:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.777020:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508920704:12345-0@lo:400:kworker.0 Request processed in 26us (4125us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760741.777026:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 917 00000100:00100000:1.0:1789760741.777035:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 918 00000100:00100000:1.0:1789760741.777039:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3481:x1876699508920448:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.777058:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3481:x1876699508920448:12345-0@lo:400:kworker.0 Request processed in 19us (4010us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760741.777061:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 918 00000100:00100000:1.0:1789760741.777093:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 919 00000100:00100000:1.0:1789760741.777097:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3481:x1876699508920960:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.777120:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3481:x1876699508920960:12345-0@lo:400:kworker.0 Request processed in 22us (4004us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760741.777126:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 919 00000100:00100000:1.0:1789760741.777137:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 920 00000100:00100000:1.0:1789760741.777142:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3481:x1876699508921472:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.777169:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3481:x1876699508921472:12345-0@lo:400:kworker.0 Request processed in 26us (4018us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760741.777180:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 920 00000100:00100000:1.0:1789760741.777206:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508921216 00000100:00100000:1.0:1789760741.777209:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508921216 00000100:00100000:1.0:1789760741.777234:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508921728 00000100:00100000:2.0:1789760741.777350:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508921344 00000100:00100000:2.0:1789760741.777355:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508921344 00000100:00100000:2.0:1789760741.777392:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 320 00000100:00100000:2.0:1789760741.777399:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508921216:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760741.777432:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508921216:12345-0@lo:400:kworker.0 Request processed in 35us (4474us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760741.777439:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 320 00000100:00100000:2.0:1789760741.777450:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 321 00000100:00100000:2.0:1789760741.777456:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fd8fd850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508921344:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760741.777482:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fd8fd850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508921344:12345-0@lo:400:kworker.0 Request processed in 26us (1030us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760741.777487:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 321 00000100:00100000:1.0:1789760741.777606:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508921728 00000100:00100000:1.0:1789760741.777675:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 322 00000100:00100000:1.0:1789760741.777681:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508921728:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.777715:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508921728:12345-0@lo:400:kworker.0 Request processed in 35us (4711us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760741.777721:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 322 00000100:00100000:0.0:1789760741.777991:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.778135:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508920320 00000100:00100000:0.0:1789760741.778140:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508920320 00000100:00100000:3.0:1789760741.778140:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484231936 00000100:00100000:3.0:1789760741.778143:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484231936 00000100:00100000:3.0:1789760741.778177:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731026840070 time 731072567852 state 0x0 ref 1 00000100:00100000:0.0:1789760741.778179:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 478 00000100:00100000:0.0:1789760741.778187:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b6c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508920320:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760741.778190:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731072828268 time 731072567852 state 0x1 ref 1 00000100:00100000:0.0:1789760741.778221:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b6c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508920320:12345-0@lo:400:kworker.0 Request processed in 35us (5024us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760741.778221:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731072828268 time 731109152576 state 0x0 ref 1 00000100:00100000:0.0:1789760741.778228:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 478 00000100:00100000:3.0:1789760741.778231:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484231936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760741.778271:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21348352, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:3.0:1789760741.778281:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.778296:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:3.0:1789760741.778300:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760741.778306:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21348352->21352447] (req 21348352->21352447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.778334:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21348352->21352447] (req 21348352->21352447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.778410:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508920192:0@lo:13:osp-pre-1-0.0 00000100:00080000:1.0:1789760741.778605:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:1.0:1789760741.778610:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:1.0:1789760741.778612:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:1.0:1789760741.778635:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c47100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508920704:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778649:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c47800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508921216:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778660:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9470e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508921728:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778670:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c45500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508920576:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778679:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c47480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508921088:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778689:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9472a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508921600:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778722:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c47b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508920320:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778732:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c45180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508920832:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778742:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c44700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508921344:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778753:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c44e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3481:1876699508920448:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778761:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c7c45880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3481:1876699508920960:0@lo:400:kworker.0 00000100:00100000:1.0:1789760741.778770:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9471880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3481:1876699508921472:0@lo:400:kworker.0 00002000:00010000:2.0:1789760741.806826:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21348352 -> 21352448 00000001:00080000:1.0:1789760741.806837:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972530 is committed 00010000:00010000:2.0:1789760741.806846:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21348352->21352447] (req 21348352->21352447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.806859:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21348352->21352447] (req 21348352->21352447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.806869:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21348352->21352447] (req 21348352->21352447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.806876:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760741.806881:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21348352->21352447] (req 21348352->21352447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.806890:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21348352->21352447] (req 21348352->21352447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.806910:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21348352->21352447] (req 21348352->21352447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.806916:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208841 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21348352->21352447] (req 21348352->21352447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760741.806994:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484231936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28758us (28999us total) trans 4294972530 rc 0/0 00000100:00100000:2.0:1789760741.807004:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5241 00000100:00100000:2.0:1789760741.807014:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731072828268 time 731109152576 state 0x2 ref 0 00000100:00100000:1.0:1789760741.814754:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.814910:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484232064 00000100:00100000:0.0:1789760741.814920:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484232064 00000100:00100000:0.0:1789760741.814987:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731072828268 time 731109152576 state 0x0 ref 1 00000100:00100000:0.0:1789760741.815002:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731109412992 time 731109152576 state 0x1 ref 1 00000100:00100000:0.0:1789760741.815118:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731109412992 time 731146047773 state 0x0 ref 1 00000100:00100000:0.0:1789760741.815134:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484232064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760741.815181:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21352448, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:0.0:1789760741.815191:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.815211:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:0.0:1789760741.815215:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760741.815221:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21352448->21356543] (req 21352448->21356543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.815234:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21352448->21356543] (req 21352448->21356543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760741.857867:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972531 is committed 00002000:00010000:0.0:1789760741.858041:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21352448 -> 21356544 00010000:00010000:0.0:1789760741.858058:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21352448->21356543] (req 21352448->21356543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.858091:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21352448->21356543] (req 21352448->21356543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.858112:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21352448->21356543] (req 21352448->21356543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.858120:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.858125:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21352448->21356543] (req 21352448->21356543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.858135:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21352448->21356543] (req 21352448->21356543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.858155:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21352448->21356543] (req 21352448->21356543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.858164:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208848 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21352448->21356543] (req 21352448->21356543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.858292:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484232064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43158us (43543us total) trans 4294972531 rc 0/0 00000100:00100000:0.0:1789760741.858302:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5242 00000100:00100000:0.0:1789760741.858322:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731109412992 time 731146047773 state 0x2 ref 0 00000100:00100000:0.0:1789760741.863727:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760741.863812:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484232192 00000100:00100000:2.0:1789760741.863821:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484232192 00000100:00100000:2.0:1789760741.863872:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731109412992 time 731146047773 state 0x0 ref 1 00000100:00100000:2.0:1789760741.863890:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731146308189 time 731146047773 state 0x1 ref 1 00000100:00100000:2.0:1789760741.863921:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731146308189 time 731194852448 state 0x0 ref 1 00000100:00100000:2.0:1789760741.863935:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484232192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760741.863984:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21356544, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:2.0:1789760741.863994:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.864010:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:2.0:1789760741.864014:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760741.864020:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21356544->21360639] (req 21356544->21360639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.864034:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21356544->21360639] (req 21356544->21360639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760741.889226:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21356544 -> 21360640 00010000:00010000:2.0:1789760741.889244:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21356544->21360639] (req 21356544->21360639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.889256:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21356544->21360639] (req 21356544->21360639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.889264:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21356544->21360639] (req 21356544->21360639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.889274:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760741.889280:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21356544->21360639] (req 21356544->21360639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.889287:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21356544->21360639] (req 21356544->21360639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.889305:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21356544->21360639] (req 21356544->21360639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760741.889322:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620884f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21356544->21360639] (req 21356544->21360639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760741.889413:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484232192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25476us (25692us total) trans 4294972532 rc 0/0 00000100:00100000:2.0:1789760741.889424:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5243 00000100:00100000:2.0:1789760741.889433:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731146308189 time 731194852448 state 0x2 ref 0 00000001:00080000:0.0:1789760741.889785:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972532 is committed 00000100:00100000:3.0:1789760741.893496:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.893731:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484232320 00000100:00100000:0.0:1789760741.893743:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484232320 00000100:00100000:0.0:1789760741.893801:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731146308189 time 731194852448 state 0x0 ref 1 00000100:00100000:0.0:1789760741.893817:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731195112864 time 731194852448 state 0x1 ref 1 00000100:00100000:0.0:1789760741.893851:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731195112864 time 731224782357 state 0x0 ref 1 00000100:00100000:0.0:1789760741.893865:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484232320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760741.893914:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21360640, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:0.0:1789760741.893923:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.893939:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:0.0:1789760741.893943:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760741.893950:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21360640->21364735] (req 21360640->21364735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.893965:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21360640->21364735] (req 21360640->21364735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760741.920521:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972533 is committed 00002000:00010000:3.0:1789760741.920640:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21360640 -> 21364736 00010000:00010000:3.0:1789760741.920656:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21360640->21364735] (req 21360640->21364735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.920670:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21360640->21364735] (req 21360640->21364735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.920678:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21360640->21364735] (req 21360640->21364735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.920686:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760741.920690:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21360640->21364735] (req 21360640->21364735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.920703:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21360640->21364735] (req 21360640->21364735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.920721:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21360640->21364735] (req 21360640->21364735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.920730:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208856 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21360640->21364735] (req 21360640->21364735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760741.920825:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484232320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26958us (27334us total) trans 4294972533 rc 0/0 00000100:00100000:3.0:1789760741.920835:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5244 00000100:00100000:3.0:1789760741.920844:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731195112864 time 731224782357 state 0x2 ref 0 00000100:00100000:1.0:1789760741.924473:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760741.924572:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484232448 00000100:00100000:3.0:1789760741.924581:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484232448 00000100:00100000:3.0:1789760741.924634:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731195112864 time 731224782357 state 0x0 ref 1 00000100:00100000:3.0:1789760741.924649:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731225042773 time 731224782357 state 0x1 ref 1 00000100:00100000:3.0:1789760741.924719:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731225042773 time 731255648236 state 0x0 ref 1 00000100:00100000:3.0:1789760741.924731:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484232448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760741.924778:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21364736, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:3.0:1789760741.924786:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.924804:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:3.0:1789760741.924809:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760741.924815:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21364736->21368831] (req 21364736->21368831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760741.924829:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21364736->21368831] (req 21364736->21368831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760741.957117:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21364736 -> 21368832 00010000:00010000:0.0:1789760741.957137:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21364736->21368831] (req 21364736->21368831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.957151:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21364736->21368831] (req 21364736->21368831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.957161:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21364736->21368831] (req 21364736->21368831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.957168:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00000001:00080000:1.0:1789760741.957171:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972534 is committed 00010000:00010000:0.0:1789760741.957173:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21364736->21368831] (req 21364736->21368831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.957181:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21364736->21368831] (req 21364736->21368831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.957199:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21364736->21368831] (req 21364736->21368831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.957208:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620885d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21364736->21368831] (req 21364736->21368831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.957762:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484232448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33027us (33365us total) trans 4294972534 rc 0/0 00000100:00100000:0.0:1789760741.957773:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5245 00000100:00100000:0.0:1789760741.957782:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731225042773 time 731255648236 state 0x2 ref 0 00000100:00100000:0.0:1789760741.961731:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760741.961798:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484232576 00000100:00100000:1.0:1789760741.961808:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484232576 00000100:00100000:1.0:1789760741.961858:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731225042773 time 731255648236 state 0x0 ref 1 00000100:00100000:1.0:1789760741.961873:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731255908652 time 731255648236 state 0x1 ref 1 00000100:00100000:1.0:1789760741.961900:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731255908652 time 731292832099 state 0x0 ref 1 00000100:00100000:1.0:1789760741.961913:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484232576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760741.961957:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21368832, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:1.0:1789760741.961965:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.961980:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:1.0:1789760741.961985:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760741.961991:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21368832->21372927] (req 21368832->21372927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760741.962005:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21368832->21372927] (req 21368832->21372927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760741.993321:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972535 is committed 00002000:00010000:0.0:1789760741.993470:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21368832 -> 21372928 00010000:00010000:0.0:1789760741.993487:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21368832->21372927] (req 21368832->21372927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.993501:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21368832->21372927] (req 21368832->21372927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.993511:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21368832->21372927] (req 21368832->21372927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.993518:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760741.993523:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21368832->21372927] (req 21368832->21372927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.993532:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21368832->21372927] (req 21368832->21372927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.993552:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21368832->21372927] (req 21368832->21372927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.993565:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208864 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21368832->21372927] (req 21368832->21372927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760741.993699:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484232576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31782us (31970us total) trans 4294972535 rc 0/0 00000100:00100000:0.0:1789760741.993710:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5246 00000100:00100000:0.0:1789760741.993720:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731255908652 time 731292832099 state 0x2 ref 0 00000100:00100000:2.0:1789760741.997812:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760741.997914:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484232704 00000100:00100000:0.0:1789760741.997924:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484232704 00000100:00100000:0.0:1789760741.997985:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731255908652 time 731292832099 state 0x0 ref 1 00000100:00100000:0.0:1789760741.998001:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731293092515 time 731292832099 state 0x1 ref 1 00000100:00100000:0.0:1789760741.998112:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731293092515 time 731329042359 state 0x0 ref 1 00000100:00100000:0.0:1789760741.998131:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484232704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760741.998180:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21372928, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:0.0:1789760741.998188:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.998203:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:0.0:1789760741.998208:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760741.998215:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21372928->21377023] (req 21372928->21377023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760741.998236:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21372928->21377023] (req 21372928->21377023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760742.040774:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972536 is committed 00002000:00010000:0.0:1789760742.040859:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21372928 -> 21377024 00010000:00010000:0.0:1789760742.040874:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21372928->21377023] (req 21372928->21377023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.040886:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21372928->21377023] (req 21372928->21377023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.040894:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21372928->21377023] (req 21372928->21377023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.040901:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760742.040906:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21372928->21377023] (req 21372928->21377023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.040915:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21372928->21377023] (req 21372928->21377023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.040934:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21372928->21377023] (req 21372928->21377023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.040941:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620886b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21372928->21377023] (req 21372928->21377023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760742.041121:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484232704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42992us (43313us total) trans 4294972536 rc 0/0 00000100:00100000:0.0:1789760742.041132:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5247 00000100:00100000:0.0:1789760742.041142:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731293092515 time 731329042359 state 0x2 ref 0 00000100:00100000:1.0:1789760742.046677:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760742.046773:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484232832 00000100:00100000:3.0:1789760742.046783:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484232832 00000100:00080000:1.0:1789760742.046837:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760742.046852:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731293092515 time 731329042359 state 0x0 ref 1 00000100:00100000:3.0:1789760742.046868:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731329302775 time 731329042359 state 0x1 ref 1 00000100:00100000:3.0:1789760742.046899:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731329302775 time 731377826200 state 0x0 ref 1 00000100:00100000:3.0:1789760742.046911:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484232832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.046957:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21377024, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:3.0:1789760742.046966:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.046981:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:3.0:1789760742.046986:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.046993:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21377024->21381119] (req 21377024->21381119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.047011:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21377024->21381119] (req 21377024->21381119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760742.125994:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972537 is committed 00002000:00010000:2.0:1789760742.126124:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21377024 -> 21381120 00010000:00010000:2.0:1789760742.126139:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21377024->21381119] (req 21377024->21381119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.126150:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21377024->21381119] (req 21377024->21381119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.126160:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21377024->21381119] (req 21377024->21381119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.126167:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760742.126172:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21377024->21381119] (req 21377024->21381119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.126180:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21377024->21381119] (req 21377024->21381119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.126200:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21377024->21381119] (req 21377024->21381119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.126208:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208872 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21377024->21381119] (req 21377024->21381119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760742.126302:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484232832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79388us (79729us total) trans 4294972537 rc 0/0 00000100:00100000:2.0:1789760742.126389:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5248 00000100:00100000:2.0:1789760742.126407:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731329302775 time 731377826200 state 0x2 ref 0 00000100:00100000:2.0:1789760742.129772:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760742.133286:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484232960 00000100:00100000:3.0:1789760742.134235:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484232960 00000100:00100000:3.0:1789760742.134371:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731329302775 time 731377826200 state 0x0 ref 1 00000100:00100000:3.0:1789760742.134387:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731378086616 time 731377826200 state 0x1 ref 1 00000100:00100000:3.0:1789760742.134411:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731378086616 time 731465343148 state 0x0 ref 1 00000100:00100000:3.0:1789760742.134424:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484232960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.134471:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21381120, mtime: 0x6aad94e5, atime: 0x0, ctime: 0x6aad94e5, blocks: 0xa807 00010000:00010000:3.0:1789760742.134480:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.134495:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:3.0:1789760742.134500:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.134506:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21381120->21385215] (req 21381120->21385215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.134519:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21381120->21385215] (req 21381120->21385215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760742.180898:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972538 is committed 00002000:00010000:2.0:1789760742.181170:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21381120 -> 21385216 00002000:00010000:2.0:1789760742.181175:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760741 -> 1789760742 00002000:00010000:2.0:1789760742.181178:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760741 -> 1789760742 00010000:00010000:2.0:1789760742.181192:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21381120->21385215] (req 21381120->21385215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.181205:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21381120->21385215] (req 21381120->21385215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.181213:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21381120->21385215] (req 21381120->21385215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.181221:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760742.181226:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21381120->21385215] (req 21381120->21385215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.181234:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21381120->21385215] (req 21381120->21385215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.181256:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21381120->21385215] (req 21381120->21385215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.181264:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208879 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21381120->21385215] (req 21381120->21385215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760742.181674:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484232960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47248us (51907us total) trans 4294972538 rc 0/0 00000100:00100000:2.0:1789760742.181685:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5249 00000100:00100000:2.0:1789760742.181695:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731378086616 time 731465343148 state 0x2 ref 0 00000100:00100000:3.0:1789760742.185560:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760742.185696:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484233088 00000100:00100000:0.0:1789760742.185707:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484233088 00000100:00100000:0.0:1789760742.185767:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731378086616 time 731465343148 state 0x0 ref 1 00000100:00100000:0.0:1789760742.185781:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731465603564 time 731465343148 state 0x1 ref 1 00000100:00100000:0.0:1789760742.185807:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731465603564 time 731516737640 state 0x0 ref 1 00000100:00100000:0.0:1789760742.185820:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484233088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760742.185866:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21385216, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:0.0:1789760742.185876:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.185892:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:0.0:1789760742.185897:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760742.185903:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21385216->21389311] (req 21385216->21389311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.185917:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21385216->21389311] (req 21385216->21389311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760742.250370:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972539 is committed 00002000:00010000:2.0:1789760742.250532:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21385216 -> 21389312 00010000:00010000:2.0:1789760742.250549:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21385216->21389311] (req 21385216->21389311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.250562:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21385216->21389311] (req 21385216->21389311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.250570:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21385216->21389311] (req 21385216->21389311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.250577:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760742.250583:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21385216->21389311] (req 21385216->21389311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.250592:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21385216->21389311] (req 21385216->21389311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.250611:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21385216->21389311] (req 21385216->21389311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.250618:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208880 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21385216->21389311] (req 21385216->21389311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760742.250718:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484233088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 64896us (65160us total) trans 4294972539 rc 0/0 00000100:00100000:2.0:1789760742.250729:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5250 00000100:00100000:2.0:1789760742.250741:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731465603564 time 731516737640 state 0x2 ref 0 00000100:00100000:1.0:1789760742.255909:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760742.255989:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484233216 00000100:00100000:0.0:1789760742.255999:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484233216 00000100:00100000:0.0:1789760742.256060:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731465603564 time 731516737640 state 0x0 ref 1 00000100:00100000:0.0:1789760742.256109:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731516998056 time 731516737640 state 0x1 ref 1 00000100:00100000:0.0:1789760742.256135:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731516998056 time 731587065762 state 0x0 ref 1 00000100:00100000:0.0:1789760742.256148:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484233216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760742.256197:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21389312, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:0.0:1789760742.256208:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.256223:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:0.0:1789760742.256228:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760742.256235:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21389312->21393407] (req 21389312->21393407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.256249:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21389312->21393407] (req 21389312->21393407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.285397:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:2.0:1789760742.306862:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21389312 -> 21393408 00010000:00010000:2.0:1789760742.306881:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21389312->21393407] (req 21389312->21393407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.306895:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21389312->21393407] (req 21389312->21393407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.306905:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21389312->21393407] (req 21389312->21393407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.306913:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760742.306918:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21389312->21393407] (req 21389312->21393407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.306927:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21389312->21393407] (req 21389312->21393407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.306947:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21389312->21393407] (req 21389312->21393407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.306956:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208887 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21389312->21393407] (req 21389312->21393407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760742.307041:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484233216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50892us (51138us total) trans 4294972540 rc 0/0 00000100:00100000:2.0:1789760742.307055:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5251 00000100:00100000:2.0:1789760742.307090:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731516998056 time 731587065762 state 0x2 ref 0 00000001:00080000:1.0:1789760742.308321:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972540 is committed 00000100:00100000:1.0:1789760742.311418:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760742.311576:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484233344 00000100:00100000:2.0:1789760742.311586:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484233344 00000100:00100000:2.0:1789760742.311639:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731516998056 time 731587065762 state 0x0 ref 1 00000100:00100000:2.0:1789760742.311654:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731587326178 time 731587065762 state 0x1 ref 1 00000100:00100000:2.0:1789760742.311680:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731587326178 time 731642611098 state 0x0 ref 1 00000100:00100000:2.0:1789760742.311692:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484233344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760742.311756:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21393408, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:2.0:1789760742.311766:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.311781:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:2.0:1789760742.311786:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760742.311793:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21393408->21397503] (req 21393408->21397503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.311808:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21393408->21397503] (req 21393408->21397503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760742.364348:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972541 is committed 00002000:00010000:2.0:1789760742.364524:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21393408 -> 21397504 00010000:00010000:2.0:1789760742.364539:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21393408->21397503] (req 21393408->21397503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.364551:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21393408->21397503] (req 21393408->21397503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.364561:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21393408->21397503] (req 21393408->21397503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.364568:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760742.364573:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21393408->21397503] (req 21393408->21397503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.364581:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21393408->21397503] (req 21393408->21397503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.364600:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21393408->21397503] (req 21393408->21397503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.364607:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620888e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21393408->21397503] (req 21393408->21397503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760742.364724:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484233344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53031us (53312us total) trans 4294972541 rc 0/0 00000100:00100000:2.0:1789760742.364735:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5252 00000100:00100000:2.0:1789760742.364745:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731587326178 time 731642611098 state 0x2 ref 0 00000100:00100000:2.0:1789760742.372253:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760742.372358:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484233472 00000100:00100000:3.0:1789760742.372367:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484233472 00000100:00100000:3.0:1789760742.372418:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731587326178 time 731642611098 state 0x0 ref 1 00000100:00100000:3.0:1789760742.372434:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731642871514 time 731642611098 state 0x1 ref 1 00000100:00100000:3.0:1789760742.372462:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731642871514 time 731703393368 state 0x0 ref 1 00000100:00100000:3.0:1789760742.372475:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484233472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.372520:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21397504, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760742.372529:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.372545:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:3.0:1789760742.372549:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.372556:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21397504->21401599] (req 21397504->21401599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.372570:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21397504->21401599] (req 21397504->21401599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760742.428105:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972542 is committed 00002000:00010000:3.0:1789760742.428298:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21397504 -> 21401600 00010000:00010000:3.0:1789760742.428330:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21397504->21401599] (req 21397504->21401599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.428344:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21397504->21401599] (req 21397504->21401599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.428354:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21397504->21401599] (req 21397504->21401599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.428362:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760742.428368:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21397504->21401599] (req 21397504->21401599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.428377:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21397504->21401599] (req 21397504->21401599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.428397:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21397504->21401599] (req 21397504->21401599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.428405:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208895 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21397504->21401599] (req 21397504->21401599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760742.428497:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484233472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56019us (56250us total) trans 4294972542 rc 0/0 00000100:00100000:3.0:1789760742.428507:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5253 00000100:00100000:3.0:1789760742.428517:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731642871514 time 731703393368 state 0x2 ref 0 00000100:00100000:0.0:1789760742.436581:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760742.436820:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484233600 00000100:00100000:3.0:1789760742.436831:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484233600 00000100:00100000:3.0:1789760742.436884:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731642871514 time 731703393368 state 0x0 ref 1 00000100:00100000:3.0:1789760742.436898:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731703653784 time 731703393368 state 0x1 ref 1 00000100:00100000:3.0:1789760742.436976:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731703653784 time 731767906113 state 0x0 ref 1 00000100:00100000:3.0:1789760742.436989:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484233600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.437034:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21401600, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760742.437044:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.437060:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:3.0:1789760742.437105:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.437112:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21401600->21405695] (req 21401600->21405695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.437127:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21401600->21405695] (req 21401600->21405695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760742.475097:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972543 is committed 00002000:00010000:2.0:1789760742.475243:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21401600 -> 21405696 00010000:00010000:2.0:1789760742.475258:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21401600->21405695] (req 21401600->21405695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.475271:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21401600->21405695] (req 21401600->21405695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.475279:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21401600->21405695] (req 21401600->21405695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.475286:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760742.475291:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21401600->21405695] (req 21401600->21405695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.475299:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21401600->21405695] (req 21401600->21405695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.475335:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21401600->21405695] (req 21401600->21405695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.475343:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620889c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21401600->21405695] (req 21401600->21405695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760742.475436:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484233600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38445us (38859us total) trans 4294972543 rc 0/0 00000100:00100000:2.0:1789760742.475446:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5254 00000100:00100000:2.0:1789760742.475455:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731703653784 time 731767906113 state 0x2 ref 0 00000100:00100000:2.0:1789760742.486544:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760742.486616:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484233728 00000100:00100000:0.0:1789760742.486627:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484233728 00000100:00100000:0.0:1789760742.486754:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731703653784 time 731767906113 state 0x0 ref 1 00000100:00100000:0.0:1789760742.486771:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731768166529 time 731767906113 state 0x1 ref 1 00000100:00100000:0.0:1789760742.486804:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731768166529 time 731817734138 state 0x0 ref 1 00000100:00100000:0.0:1789760742.486829:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484233728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760742.486874:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21405696, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:0.0:1789760742.486882:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.486897:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:0.0:1789760742.486901:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760742.486907:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21405696->21409791] (req 21405696->21409791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.486921:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21405696->21409791] (req 21405696->21409791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760742.530456:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972544 is committed 00002000:00010000:2.0:1789760742.530593:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21405696 -> 21409792 00010000:00010000:2.0:1789760742.530610:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21405696->21409791] (req 21405696->21409791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.530757:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21405696->21409791] (req 21405696->21409791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.530767:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21405696->21409791] (req 21405696->21409791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.530775:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760742.530780:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21405696->21409791] (req 21405696->21409791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.530790:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21405696->21409791] (req 21405696->21409791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.530813:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21405696->21409791] (req 21405696->21409791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.530822:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962088a3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21405696->21409791] (req 21405696->21409791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760742.530961:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484233728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44140us (44420us total) trans 4294972544 rc 0/0 00000100:00100000:2.0:1789760742.530975:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5255 00000100:00100000:2.0:1789760742.530986:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731768166529 time 731817734138 state 0x2 ref 0 00000100:00100000:0.0:1789760742.538880:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760742.542621:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484233856 00000100:00100000:2.0:1789760742.542636:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484233856 00000100:00100000:3.0:1789760742.542786:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731768166529 time 731817734138 state 0x0 ref 1 00000100:00100000:3.0:1789760742.542801:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731817994554 time 731817734138 state 0x1 ref 1 00000100:00100000:3.0:1789760742.542833:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731817994554 time 731873764269 state 0x0 ref 1 00000100:00100000:3.0:1789760742.542846:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484233856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.542891:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21409792, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760742.542900:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.542916:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:3.0:1789760742.542921:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.542927:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21409792->21413887] (req 21409792->21413887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.542940:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21409792->21413887] (req 21409792->21413887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760742.575197:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972545 is committed 00002000:00010000:2.0:1789760742.575350:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21409792 -> 21413888 00010000:00010000:2.0:1789760742.575364:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21409792->21413887] (req 21409792->21413887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.575378:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21409792->21413887] (req 21409792->21413887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.575387:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21409792->21413887] (req 21409792->21413887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.575394:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760742.575398:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21409792->21413887] (req 21409792->21413887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.575407:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21409792->21413887] (req 21409792->21413887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.575427:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21409792->21413887] (req 21409792->21413887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.575434:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962088aa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21409792->21413887] (req 21409792->21413887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760742.575526:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484233856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32677us (36651us total) trans 4294972545 rc 0/0 00000100:00100000:2.0:1789760742.575537:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5256 00000100:00100000:2.0:1789760742.575548:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731817994554 time 731873764269 state 0x2 ref 0 00000100:00100000:1.0:1789760742.580871:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760742.580931:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484233984 00000100:00100000:3.0:1789760742.580946:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484233984 00000100:00100000:3.0:1789760742.580997:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731817994554 time 731873764269 state 0x0 ref 1 00000100:00100000:3.0:1789760742.581012:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731874024685 time 731873764269 state 0x1 ref 1 00000100:00100000:3.0:1789760742.581043:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731874024685 time 731911974355 state 0x0 ref 1 00000100:00100000:3.0:1789760742.581055:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484233984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.581131:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21413888, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760742.581140:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.581156:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:3.0:1789760742.581160:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.581167:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21413888->21417983] (req 21413888->21417983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.581186:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21413888->21417983] (req 21413888->21417983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760742.622501:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972546 is committed 00002000:00010000:2.0:1789760742.622588:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21413888 -> 21417984 00010000:00010000:2.0:1789760742.622603:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21413888->21417983] (req 21413888->21417983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.622617:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21413888->21417983] (req 21413888->21417983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.622626:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21413888->21417983] (req 21413888->21417983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.622633:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760742.622639:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21413888->21417983] (req 21413888->21417983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.622648:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21413888->21417983] (req 21413888->21417983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.622668:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21413888->21417983] (req 21413888->21417983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.622804:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962088b1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21413888->21417983] (req 21413888->21417983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760742.622930:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484233984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41871us (42061us total) trans 4294972546 rc 0/0 00000100:00100000:2.0:1789760742.622940:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5257 00000100:00100000:2.0:1789760742.622949:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731874024685 time 731911974355 state 0x2 ref 0 00000100:00100000:1.0:1789760742.629375:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760742.629468:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484234112 00000100:00100000:2.0:1789760742.629478:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484234112 00000100:00100000:2.0:1789760742.629533:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731874024685 time 731911974355 state 0x0 ref 1 00000100:00100000:2.0:1789760742.629549:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731912234771 time 731911974355 state 0x1 ref 1 00000100:00100000:2.0:1789760742.629582:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731912234771 time 731960513430 state 0x0 ref 1 00000100:00100000:2.0:1789760742.629596:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484234112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760742.629646:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21417984, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:2.0:1789760742.629655:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.629672:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:2.0:1789760742.629676:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760742.629683:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21417984->21422079] (req 21417984->21422079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.629704:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21417984->21422079] (req 21417984->21422079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760742.633480:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760742.633520:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760742.633550:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760742.633570:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760742.633626:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760742.633656:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484234240 00000100:00100000:1.0:1789760742.633665:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484234240 00000100:00100000:1.0:1789760742.633713:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 479 00000100:00100000:1.0:1789760742.633723:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b5c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484234240:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760742.633787:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b5c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484234240:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 65us (233us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760742.633794:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 479 00000100:00100000:1.0:1789760742.633824:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484234368 00000100:00100000:1.0:1789760742.633828:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484234368 00000100:00100000:1.0:1789760742.633858:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484234624 00000100:00100000:1.0:1789760742.633861:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484234624 00000100:00100000:1.0:1789760742.633877:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484234496 00000100:00100000:1.0:1789760742.633879:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484234496 00000100:00100000:1.0:1789760742.633889:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484234752 00000100:00100000:1.0:1789760742.633891:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484234752 00000100:00100000:1.0:1789760742.633905:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 921 00000100:00100000:1.0:1789760742.633911:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484234368:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760742.633941:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484234368:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 30us (472us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760742.633947:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 921 00000100:00100000:1.0:1789760742.633959:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 922 00000100:00100000:1.0:1789760742.633966:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484234624:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760742.633987:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484234624:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 21us (466us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760742.633993:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 922 00000100:00100000:1.0:1789760742.634003:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 923 00000100:00100000:1.0:1789760742.634009:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484234496:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760742.634032:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484234496:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 23us (461us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760742.634038:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 923 00000100:00100000:1.0:1789760742.634048:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 924 00000100:00100000:1.0:1789760742.634054:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484234752:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760742.634099:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484234752:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 45us (473us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760742.634105:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 924 00000100:00080000:1.0:1789760742.634124:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:1.0:1789760742.634128:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000001:00080000:2.0:1789760742.658857:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972547 is committed 00002000:00010000:1.0:1789760742.660228:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21417984 -> 21422080 00010000:00010000:1.0:1789760742.660244:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21417984->21422079] (req 21417984->21422079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760742.660258:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21417984->21422079] (req 21417984->21422079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760742.660267:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21417984->21422079] (req 21417984->21422079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760742.660281:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760742.660286:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21417984->21422079] (req 21417984->21422079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760742.660296:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21417984->21422079] (req 21417984->21422079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760742.660324:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21417984->21422079] (req 21417984->21422079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760742.660332:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088b8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21417984->21422079] (req 21417984->21422079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760742.660485:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484234112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30886us (31113us total) trans 4294972547 rc 0/0 00000100:00100000:1.0:1789760742.660495:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5258 00000100:00100000:1.0:1789760742.660505:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731912234771 time 731960513430 state 0x2 ref 0 00000100:00100000:1.0:1789760742.667248:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760742.667626:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484234880 00000100:00100000:2.0:1789760742.667637:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484234880 00000100:00100000:2.0:1789760742.667689:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731912234771 time 731960513430 state 0x0 ref 1 00000100:00100000:2.0:1789760742.667705:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731960773846 time 731960513430 state 0x1 ref 1 00000100:00100000:2.0:1789760742.667775:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731960773846 time 731998705858 state 0x0 ref 1 00000100:00100000:2.0:1789760742.667788:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484234880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760742.667833:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21422080, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:2.0:1789760742.667842:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.667858:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=731s v=(0 0) 00010000:00010000:2.0:1789760742.667862:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760742.667868:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21422080->21426175] (req 21422080->21426175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.667882:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21422080->21426175] (req 21422080->21426175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760742.697560:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972548 is committed 00002000:00010000:3.0:1789760742.697670:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21422080 -> 21426176 00010000:00010000:3.0:1789760742.697795:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21422080->21426175] (req 21422080->21426175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.697805:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21422080->21426175] (req 21422080->21426175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.697811:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21422080->21426175] (req 21422080->21426175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.697815:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760742.697819:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21422080->21426175] (req 21422080->21426175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.697825:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21422080->21426175] (req 21422080->21426175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.697841:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21422080->21426175] (req 21422080->21426175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.697846:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962088bf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21422080->21426175] (req 21422080->21426175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760742.697930:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484234880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30139us (30689us total) trans 4294972548 rc 0/0 00000100:00100000:3.0:1789760742.697941:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5259 00000100:00100000:3.0:1789760742.697950:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731960773846 time 731998705858 state 0x2 ref 0 00000100:00100000:2.0:1789760742.702561:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760742.702721:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484235008 00000100:00100000:3.0:1789760742.702730:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484235008 00000100:00100000:3.0:1789760742.702783:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731960773846 time 731998705858 state 0x0 ref 1 00000100:00100000:3.0:1789760742.702798:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731998966274 time 731998705858 state 0x1 ref 1 00000100:00100000:3.0:1789760742.702830:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731998966274 time 732033760803 state 0x0 ref 1 00000100:00100000:3.0:1789760742.702842:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484235008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.702889:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21426176, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760742.702898:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.702914:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760742.702918:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.702924:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21426176->21430271] (req 21426176->21430271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.702938:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21426176->21430271] (req 21426176->21430271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760742.735355:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21426176 -> 21430272 00010000:00010000:0.0:1789760742.735373:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21426176->21430271] (req 21426176->21430271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.735384:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21426176->21430271] (req 21426176->21430271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.735393:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21426176->21430271] (req 21426176->21430271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.735400:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760742.735751:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21426176->21430271] (req 21426176->21430271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.735771:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21426176->21430271] (req 21426176->21430271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.735791:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21426176->21430271] (req 21426176->21430271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.735800:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962088c6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21426176->21430271] (req 21426176->21430271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760742.736102:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484235008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33221us (33510us total) trans 4294972549 rc 0/0 00000100:00100000:0.0:1789760742.736114:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5260 00000100:00100000:0.0:1789760742.736123:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731998966274 time 732033760803 state 0x2 ref 0 00000001:00080000:0.0:1789760742.736174:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972549 is committed 00000100:00100000:1.0:1789760742.742707:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760742.742904:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484235136 00000100:00100000:3.0:1789760742.742914:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484235136 00000100:00100000:3.0:1789760742.742967:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 731998966274 time 732033760803 state 0x0 ref 1 00000100:00100000:3.0:1789760742.742981:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732034021219 time 732033760803 state 0x1 ref 1 00000100:00100000:3.0:1789760742.743062:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732034021219 time 732073992337 state 0x0 ref 1 00000100:00100000:3.0:1789760742.743114:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484235136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.743172:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21430272, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760742.743180:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.743196:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760742.743201:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.743206:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21430272->21434367] (req 21430272->21434367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.743219:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21430272->21434367] (req 21430272->21434367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760742.773651:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972550 is committed 00002000:00010000:3.0:1789760742.773801:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21430272 -> 21434368 00010000:00010000:3.0:1789760742.773818:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21430272->21434367] (req 21430272->21434367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.773837:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21430272->21434367] (req 21430272->21434367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.773847:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21430272->21434367] (req 21430272->21434367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.773854:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760742.773859:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21430272->21434367] (req 21430272->21434367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.773868:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21430272->21434367] (req 21430272->21434367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.773888:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21430272->21434367] (req 21430272->21434367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.773896:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088cd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21430272->21434367] (req 21430272->21434367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760742.774144:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484235136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31028us (31442us total) trans 4294972550 rc 0/0 00000100:00100000:3.0:1789760742.774154:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5261 00000100:00100000:3.0:1789760742.774164:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732034021219 time 732073992337 state 0x2 ref 0 00000100:00100000:1.0:1789760742.777852:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760742.777930:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484235264 00000100:00100000:2.0:1789760742.777939:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484235264 00000100:00100000:2.0:1789760742.777997:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732034021219 time 732073992337 state 0x0 ref 1 00000100:00100000:2.0:1789760742.778011:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732074252753 time 732073992337 state 0x1 ref 1 00000100:00100000:2.0:1789760742.778039:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732074252753 time 732108970592 state 0x0 ref 1 00000100:00100000:2.0:1789760742.778051:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484235264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760742.778130:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21434368, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:2.0:1789760742.778139:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.778156:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:2.0:1789760742.778161:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760742.778167:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21434368->21438463] (req 21434368->21438463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760742.778181:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21434368->21438463] (req 21434368->21438463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760742.823852:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972551 is committed 00002000:00010000:3.0:1789760742.824152:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21434368 -> 21438464 00010000:00010000:3.0:1789760742.824169:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21434368->21438463] (req 21434368->21438463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.824181:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21434368->21438463] (req 21434368->21438463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.824189:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21434368->21438463] (req 21434368->21438463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.824197:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760742.824202:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21434368->21438463] (req 21434368->21438463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.824211:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21434368->21438463] (req 21434368->21438463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.824231:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21434368->21438463] (req 21434368->21438463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.824239:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962088d4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21434368->21438463] (req 21434368->21438463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760742.824336:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484235264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46281us (46487us total) trans 4294972551 rc 0/0 00000100:00100000:3.0:1789760742.824346:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5262 00000100:00100000:3.0:1789760742.824356:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732074252753 time 732108970592 state 0x2 ref 0 00000100:00100000:2.0:1789760742.827998:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760742.828108:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484235392 00000100:00100000:3.0:1789760742.828118:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484235392 00000100:00100000:3.0:1789760742.828174:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732074252753 time 732108970592 state 0x0 ref 1 00000100:00100000:3.0:1789760742.828190:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732109231008 time 732108970592 state 0x1 ref 1 00000100:00100000:3.0:1789760742.828213:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732109231008 time 732159144166 state 0x0 ref 1 00000100:00100000:3.0:1789760742.828225:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484235392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.828273:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21438464, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760742.828283:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.828299:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760742.828303:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.828320:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21438464->21442559] (req 21438464->21442559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.828334:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21438464->21442559] (req 21438464->21442559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760742.893554:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972552 is committed 00002000:00010000:3.0:1789760742.893661:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21438464 -> 21442560 00010000:00010000:3.0:1789760742.893677:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21438464->21442559] (req 21438464->21442559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.893690:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21438464->21442559] (req 21438464->21442559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.893701:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21438464->21442559] (req 21438464->21442559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.893708:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760742.893712:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21438464->21442559] (req 21438464->21442559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.893722:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21438464->21442559] (req 21438464->21442559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.893852:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21438464->21442559] (req 21438464->21442559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.893862:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088db lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21438464->21442559] (req 21438464->21442559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760742.893994:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484235392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65765us (66001us total) trans 4294972552 rc 0/0 00000100:00100000:3.0:1789760742.894004:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5263 00000100:00100000:3.0:1789760742.894014:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732109231008 time 732159144166 state 0x2 ref 0 00000100:00100000:1.0:1789760742.901389:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760742.901656:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484235520 00000100:00100000:0.0:1789760742.901671:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484235520 00000100:00100000:0.0:1789760742.901736:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732109231008 time 732159144166 state 0x0 ref 1 00000100:00100000:0.0:1789760742.901751:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732159404582 time 732159144166 state 0x1 ref 1 00000100:00100000:0.0:1789760742.901781:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732159404582 time 732232711668 state 0x0 ref 1 00000100:00100000:0.0:1789760742.901795:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484235520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760742.901850:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21442560, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:0.0:1789760742.901859:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.901875:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:0.0:1789760742.901879:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760742.901885:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21442560->21446655] (req 21442560->21446655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.901898:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21442560->21446655] (req 21442560->21446655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760742.932232:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972553 is committed 00002000:00010000:3.0:1789760742.932341:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21442560 -> 21446656 00010000:00010000:3.0:1789760742.932357:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21442560->21446655] (req 21442560->21446655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.932376:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21442560->21446655] (req 21442560->21446655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.932386:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21442560->21446655] (req 21442560->21446655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.932393:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760742.932398:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21442560->21446655] (req 21442560->21446655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.932407:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21442560->21446655] (req 21442560->21446655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.932426:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21442560->21446655] (req 21442560->21446655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.932435:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962088e2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21442560->21446655] (req 21442560->21446655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760742.932536:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484235520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30739us (31152us total) trans 4294972553 rc 0/0 00000100:00100000:3.0:1789760742.932546:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5264 00000100:00100000:3.0:1789760742.932556:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732159404582 time 732232711668 state 0x2 ref 0 00000100:00100000:1.0:1789760742.935704:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760742.935949:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484235648 00000100:00100000:0.0:1789760742.935959:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484235648 00000100:00100000:0.0:1789760742.936013:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732159404582 time 732232711668 state 0x0 ref 1 00000100:00100000:0.0:1789760742.936028:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732232972084 time 732232711668 state 0x1 ref 1 00000100:00100000:0.0:1789760742.936108:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732232972084 time 732267038273 state 0x0 ref 1 00000100:00100000:0.0:1789760742.936121:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484235648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760742.936164:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21446656, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:0.0:1789760742.936172:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.936187:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:0.0:1789760742.936192:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760742.936197:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21446656->21450751] (req 21446656->21450751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.936210:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21446656->21450751] (req 21446656->21450751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760742.968372:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21446656 -> 21450752 00010000:00010000:0.0:1789760742.968405:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21446656->21450751] (req 21446656->21450751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.968420:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21446656->21450751] (req 21446656->21450751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.968430:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21446656->21450751] (req 21446656->21450751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.968437:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760742.968441:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21446656->21450751] (req 21446656->21450751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.968451:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21446656->21450751] (req 21446656->21450751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.968470:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21446656->21450751] (req 21446656->21450751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760742.968478:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962088e9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21446656->21450751] (req 21446656->21450751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760742.968579:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484235648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32456us (32880us total) trans 4294972554 rc 0/0 00000100:00100000:0.0:1789760742.968590:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5265 00000100:00100000:0.0:1789760742.968599:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732232972084 time 732267038273 state 0x2 ref 0 00000001:00080000:2.0:1789760742.969514:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972554 is committed 00000100:00100000:2.0:1789760742.972227:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760742.972319:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484235776 00000100:00100000:3.0:1789760742.972330:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484235776 00000100:00100000:3.0:1789760742.972397:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732232972084 time 732267038273 state 0x0 ref 1 00000100:00100000:3.0:1789760742.972422:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732267298689 time 732267038273 state 0x1 ref 1 00000100:00100000:3.0:1789760742.972453:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732267298689 time 732303384105 state 0x0 ref 1 00000100:00100000:3.0:1789760742.972467:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484235776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760742.972516:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21450752, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760742.972525:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.972541:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760742.972545:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760742.972551:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21450752->21454847] (req 21450752->21454847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760742.972565:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21450752->21454847] (req 21450752->21454847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760743.019718:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972555 is committed 00002000:00010000:1.0:1789760743.019981:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21450752 -> 21454848 00010000:00010000:1.0:1789760743.019997:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21450752->21454847] (req 21450752->21454847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.020011:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21450752->21454847] (req 21450752->21454847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.020021:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21450752->21454847] (req 21450752->21454847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.020556:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760743.020561:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21450752->21454847] (req 21450752->21454847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.020571:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21450752->21454847] (req 21450752->21454847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.020593:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21450752->21454847] (req 21450752->21454847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.020601:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962088f0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21450752->21454847] (req 21450752->21454847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760743.020835:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484235776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48366us (48612us total) trans 4294972555 rc 0/0 00000100:00100000:1.0:1789760743.020845:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5266 00000100:00100000:1.0:1789760743.020856:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732267298689 time 732303384105 state 0x2 ref 0 00000100:00100000:3.0:1789760743.026506:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760743.027021:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484235904 00000100:00100000:2.0:1789760743.027031:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484235904 00000100:00100000:2.0:1789760743.027137:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732267298689 time 732303384105 state 0x0 ref 1 00000100:00100000:2.0:1789760743.027152:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732303644521 time 732303384105 state 0x1 ref 1 00000100:00100000:2.0:1789760743.027174:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732303644521 time 732358104839 state 0x0 ref 1 00000100:00100000:2.0:1789760743.027191:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484235904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760743.027241:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21454848, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:2.0:1789760743.027250:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.027264:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:2.0:1789760743.027268:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760743.027274:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21454848->21458943] (req 21454848->21458943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.027287:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21454848->21458943] (req 21454848->21458943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760743.072349:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21454848 -> 21458944 00010000:00010000:2.0:1789760743.072369:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21454848->21458943] (req 21454848->21458943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.072389:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21454848->21458943] (req 21454848->21458943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.072398:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21454848->21458943] (req 21454848->21458943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.072406:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760743.072411:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21454848->21458943] (req 21454848->21458943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.072420:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21454848->21458943] (req 21454848->21458943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.072439:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21454848->21458943] (req 21454848->21458943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.072446:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962088f7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21454848->21458943] (req 21454848->21458943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760743.072546:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484235904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45351us (46044us total) trans 4294972556 rc 0/0 00000100:00100000:2.0:1789760743.072557:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5267 00000100:00100000:2.0:1789760743.072567:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732303644521 time 732358104839 state 0x2 ref 0 00000001:00080000:1.0:1789760743.073511:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972556 is committed 00000100:00100000:0.0:1789760743.080618:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.080750:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484236032 00000100:00100000:3.0:1789760743.080769:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484236032 00000100:00100000:3.0:1789760743.080821:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732303644521 time 732358104839 state 0x0 ref 1 00000100:00100000:3.0:1789760743.080837:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732358365255 time 732358104839 state 0x1 ref 1 00000100:00100000:3.0:1789760743.080868:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732358365255 time 732411798818 state 0x0 ref 1 00000100:00100000:3.0:1789760743.080889:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.080937:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21458944, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760743.080945:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.080961:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760743.080967:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.080973:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21458944->21463039] (req 21458944->21463039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.080987:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21458944->21463039] (req 21458944->21463039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760743.130786:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21458944 -> 21463040 00010000:00010000:1.0:1789760743.130807:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21458944->21463039] (req 21458944->21463039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.130826:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21458944->21463039] (req 21458944->21463039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.130835:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21458944->21463039] (req 21458944->21463039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.130842:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760743.130847:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21458944->21463039] (req 21458944->21463039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.130856:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21458944->21463039] (req 21458944->21463039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.130875:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21458944->21463039] (req 21458944->21463039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.130888:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962088fe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21458944->21463039] (req 21458944->21463039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760743.130980:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484236032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50096us (50376us total) trans 4294972557 rc 0/0 00000100:00100000:1.0:1789760743.130994:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5268 00000100:00100000:1.0:1789760743.131003:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732358365255 time 732411798818 state 0x2 ref 0 00000001:00080000:1.0:1789760743.131044:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972557 is committed 00000100:00100000:2.0:1789760743.136599:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.136677:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484236160 00000100:00100000:3.0:1789760743.136686:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484236160 00000100:00100000:3.0:1789760743.136742:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732358365255 time 732411798818 state 0x0 ref 1 00000100:00100000:3.0:1789760743.136756:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732412059234 time 732411798818 state 0x1 ref 1 00000100:00100000:3.0:1789760743.136786:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732412059234 time 732467716801 state 0x0 ref 1 00000100:00100000:3.0:1789760743.136799:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706d050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.136845:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21463040, mtime: 0x6aad94e6, atime: 0x0, ctime: 0x6aad94e6, blocks: 0xa807 00010000:00010000:3.0:1789760743.136854:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.136874:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760743.136878:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.136884:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21463040->21467135] (req 21463040->21467135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.136898:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21463040->21467135] (req 21463040->21467135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760743.163741:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972558 is committed 00002000:00010000:3.0:1789760743.163843:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21463040 -> 21467136 00002000:00010000:3.0:1789760743.163848:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760742 -> 1789760743 00002000:00010000:3.0:1789760743.163851:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760742 -> 1789760743 00010000:00010000:3.0:1789760743.163863:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21463040->21467135] (req 21463040->21467135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.163877:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21463040->21467135] (req 21463040->21467135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.163886:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21463040->21467135] (req 21463040->21467135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.163895:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.163900:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21463040->21467135] (req 21463040->21467135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.163909:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21463040->21467135] (req 21463040->21467135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.163928:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21463040->21467135] (req 21463040->21467135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.163935:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208905 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21463040->21467135] (req 21463040->21467135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.164029:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706d050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27228us (27433us total) trans 4294972558 rc 0/0 00000100:00100000:3.0:1789760743.164038:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5269 00000100:00100000:3.0:1789760743.164047:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732412059234 time 732467716801 state 0x2 ref 0 00000100:00100000:2.0:1789760743.167292:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760743.167395:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484236288 00000100:00100000:1.0:1789760743.167405:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484236288 00000100:00100000:1.0:1789760743.167462:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732412059234 time 732467716801 state 0x0 ref 1 00000100:00100000:1.0:1789760743.167480:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732467977217 time 732467716801 state 0x1 ref 1 00000100:00100000:1.0:1789760743.167509:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732467977217 time 732498440251 state 0x0 ref 1 00000100:00100000:1.0:1789760743.167522:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760743.167577:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21467136, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:1.0:1789760743.167586:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.167601:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:1.0:1789760743.167606:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760743.167613:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21467136->21471231] (req 21467136->21471231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760743.167627:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21467136->21471231] (req 21467136->21471231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760743.216668:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972559 is committed 00002000:00010000:2.0:1789760743.217019:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21467136 -> 21471232 00010000:00010000:2.0:1789760743.217034:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21467136->21471231] (req 21467136->21471231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.217046:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21467136->21471231] (req 21467136->21471231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.217055:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21467136->21471231] (req 21467136->21471231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.217062:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760743.217159:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21467136->21471231] (req 21467136->21471231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.217170:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21467136->21471231] (req 21467136->21471231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.217190:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21467136->21471231] (req 21467136->21471231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.217198:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620890c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21467136->21471231] (req 21467136->21471231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760743.217293:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49768us (50005us total) trans 4294972559 rc 0/0 00000100:00100000:2.0:1789760743.217314:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5270 00000100:00100000:2.0:1789760743.217324:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732467977217 time 732498440251 state 0x2 ref 0 00000100:00100000:0.0:1789760743.221520:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760743.221572:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484236416 00000100:00100000:2.0:1789760743.221581:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484236416 00000100:00100000:2.0:1789760743.221645:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732467977217 time 732498440251 state 0x0 ref 1 00000100:00100000:2.0:1789760743.221660:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732498700667 time 732498440251 state 0x1 ref 1 00000100:00100000:2.0:1789760743.221689:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732498700667 time 732552619800 state 0x0 ref 1 00000100:00100000:2.0:1789760743.221703:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760743.221762:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21471232, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:2.0:1789760743.221771:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.221786:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:2.0:1789760743.221791:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760743.221797:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21471232->21475327] (req 21471232->21475327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.221822:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21471232->21475327] (req 21471232->21475327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760743.269316:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21471232 -> 21475328 00010000:00010000:0.0:1789760743.269338:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21471232->21475327] (req 21471232->21475327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.269352:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21471232->21475327] (req 21471232->21475327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.269362:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21471232->21475327] (req 21471232->21475327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.269369:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760743.269374:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21471232->21475327] (req 21471232->21475327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.269384:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21471232->21475327] (req 21471232->21475327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.269476:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21471232->21475327] (req 21471232->21475327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.269485:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208913 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21471232->21475327] (req 21471232->21475327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760743.269578:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484236416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47873us (48063us total) trans 4294972560 rc 0/0 00000100:00100000:0.0:1789760743.269589:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5271 00000100:00100000:0.0:1789760743.269600:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732498700667 time 732552619800 state 0x2 ref 0 00000001:00080000:0.0:1789760743.269636:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972560 is committed 00000100:00100000:0.0:1789760743.278003:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.278112:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484236544 00000100:00100000:3.0:1789760743.278122:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484236544 00000100:00100000:3.0:1789760743.278175:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732498700667 time 732552619800 state 0x0 ref 1 00000100:00100000:3.0:1789760743.278190:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732552880216 time 732552619800 state 0x1 ref 1 00000100:00100000:3.0:1789760743.278222:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732552880216 time 732609152006 state 0x0 ref 1 00000100:00100000:3.0:1789760743.278235:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.278282:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21475328, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.278291:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.278317:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760743.278323:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.278329:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21475328->21479423] (req 21475328->21479423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.278343:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21475328->21479423] (req 21475328->21479423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760743.303809:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21475328 -> 21479424 00010000:00010000:3.0:1789760743.303829:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21475328->21479423] (req 21475328->21479423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.303841:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21475328->21479423] (req 21475328->21479423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.303849:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21475328->21479423] (req 21475328->21479423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.303857:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.303862:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21475328->21479423] (req 21475328->21479423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.303871:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21475328->21479423] (req 21475328->21479423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.303890:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21475328->21479423] (req 21475328->21479423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.303897:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620891a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21475328->21479423] (req 21475328->21479423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.303989:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484236544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25751us (25990us total) trans 4294972561 rc 0/0 00000100:00100000:3.0:1789760743.304000:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5272 00000100:00100000:3.0:1789760743.304010:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732552880216 time 732609152006 state 0x2 ref 0 00000001:00080000:0.0:1789760743.306109:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972561 is committed 00010000:00010000:3.0:1789760743.308355:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760743.308688:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.308819:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484236672 00000100:00100000:3.0:1789760743.308827:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484236672 00000100:00100000:3.0:1789760743.308878:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732552880216 time 732609152006 state 0x0 ref 1 00000100:00100000:3.0:1789760743.308894:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732609412422 time 732609152006 state 0x1 ref 1 00000100:00100000:3.0:1789760743.308919:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732609412422 time 732639849620 state 0x0 ref 1 00000100:00100000:3.0:1789760743.308931:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.308975:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21479424, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.308983:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.308998:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760743.309002:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.309009:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21479424->21483519] (req 21479424->21483519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.309022:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21479424->21483519] (req 21479424->21483519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.309387:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1789760743.309465:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8910000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508921984:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760743.309513:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760743.309572:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508921984 00000100:00100000:3.0:1789760743.309576:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508921984 00000100:00100000:3.0:1789760743.309607:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 382 00000100:00100000:3.0:1789760743.309613:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508921984:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:3.0:1789760743.309626:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527499264 pending 1703936 free 3836739584 avail 3798990848 00000100:00100000:3.0:1789760743.309662:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508921984:12345-0@lo:13:osp-pre-0-0.0 Request processed in 48us (147us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760743.309668:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 382 00000100:00080000:3.0:1789760743.309684:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760743.309713:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8910000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508921984:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760743.309886:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760743.309895:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508922112:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760743.309926:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760743.309957:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508922112 00000100:00100000:0.0:1789760743.309960:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508922112 00000100:00100000:0.0:1789760743.309989:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 441 00000100:00100000:0.0:1789760743.309995:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508922112:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:0.0:1789760743.310023:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:0.0:1789760743.310051:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508922112:12345-0@lo:41:osp-pre-0-1.0 Request processed in 56us (125us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760743.310057:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 441 00000100:00100000:0.0:1789760743.310117:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508922112:0@lo:41:osp-pre-0-1.0 00000100:00100000:2.0:1789760743.314026:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760743.314050:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508922240:0@lo:13:osp-pre-0-1.0 00000100:00100000:2.0:1789760743.314144:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760743.314651:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508922240 00000100:00100000:1.0:1789760743.314659:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508922240 00000100:00100000:1.0:1789760743.314722:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 383 00000100:00100000:1.0:1789760743.314732:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508922240:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:1.0:1789760743.314755:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527499264 pending 1703936 free 3836739584 avail 3798990848 00000100:00100000:1.0:1789760743.314820:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508922240:12345-0@lo:13:osp-pre-0-1.0 Request processed in 89us (675us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760743.314828:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 383 00000100:00100000:1.0:1789760743.314876:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508922240:0@lo:13:osp-pre-0-1.0 00000100:00080000:3.0:1789760743.315859:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:3.0:1789760743.341917:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21479424 -> 21483520 00010000:00010000:3.0:1789760743.341939:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21479424->21483519] (req 21479424->21483519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.341951:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21479424->21483519] (req 21479424->21483519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.341960:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21479424->21483519] (req 21479424->21483519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.341967:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.341973:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21479424->21483519] (req 21479424->21483519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.341982:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21479424->21483519] (req 21479424->21483519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.342005:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21479424->21483519] (req 21479424->21483519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.342013:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208921 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21479424->21483519] (req 21479424->21483519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.342142:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484236672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33207us (33455us total) trans 4294972562 rc 0/0 00000100:00100000:3.0:1789760743.342153:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5273 00000100:00100000:3.0:1789760743.342163:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732609412422 time 732639849620 state 0x2 ref 0 00000001:00080000:0.0:1789760743.342366:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972562 is committed 00000100:00100000:2.0:1789760743.345503:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.345645:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484236800 00000100:00100000:3.0:1789760743.345655:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484236800 00000100:00100000:3.0:1789760743.345708:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732609412422 time 732639849620 state 0x0 ref 1 00000100:00100000:3.0:1789760743.345723:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732640110036 time 732639849620 state 0x1 ref 1 00000100:00100000:3.0:1789760743.345756:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732640110036 time 732676687398 state 0x0 ref 1 00000100:00100000:3.0:1789760743.345769:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.345816:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21483520, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.345825:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.345840:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760743.345845:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.345851:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21483520->21487615] (req 21483520->21487615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.345865:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21483520->21487615] (req 21483520->21487615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760743.375808:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972563 is committed 00002000:00010000:3.0:1789760743.375897:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21483520 -> 21487616 00010000:00010000:3.0:1789760743.375911:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21483520->21487615] (req 21483520->21487615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.375924:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21483520->21487615] (req 21483520->21487615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.375933:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21483520->21487615] (req 21483520->21487615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.375939:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.375944:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21483520->21487615] (req 21483520->21487615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.375952:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21483520->21487615] (req 21483520->21487615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.375979:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21483520->21487615] (req 21483520->21487615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.375987:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208928 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21483520->21487615] (req 21483520->21487615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.376225:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484236800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30452us (30726us total) trans 4294972563 rc 0/0 00000100:00100000:3.0:1789760743.376236:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5274 00000100:00100000:3.0:1789760743.376245:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732640110036 time 732676687398 state 0x2 ref 0 00000100:00100000:0.0:1789760743.379689:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.379739:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484236928 00000100:00100000:3.0:1789760743.379749:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484236928 00000100:00100000:3.0:1789760743.379802:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732640110036 time 732676687398 state 0x0 ref 1 00000100:00100000:3.0:1789760743.379817:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732676947814 time 732676687398 state 0x1 ref 1 00000100:00100000:3.0:1789760743.379849:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732676947814 time 732710779648 state 0x0 ref 1 00000100:00100000:3.0:1789760743.379860:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484236928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.379907:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21487616, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.379916:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.379931:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760743.379937:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.379943:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21487616->21491711] (req 21487616->21491711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.379958:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21487616->21491711] (req 21487616->21491711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760743.409953:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972564 is committed 00002000:00010000:3.0:1789760743.409953:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21487616 -> 21491712 00010000:00010000:3.0:1789760743.409970:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21487616->21491711] (req 21487616->21491711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.409983:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21487616->21491711] (req 21487616->21491711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.409992:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21487616->21491711] (req 21487616->21491711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.409999:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.410005:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21487616->21491711] (req 21487616->21491711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.410014:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21487616->21491711] (req 21487616->21491711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.410034:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21487616->21491711] (req 21487616->21491711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.410042:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620892f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21487616->21491711] (req 21487616->21491711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.410353:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484236928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30489us (30674us total) trans 4294972564 rc 0/0 00000100:00100000:3.0:1789760743.410365:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5275 00000100:00100000:3.0:1789760743.410378:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732676947814 time 732710779648 state 0x2 ref 0 00000100:00100000:2.0:1789760743.416518:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.416714:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484237056 00000100:00100000:3.0:1789760743.416724:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484237056 00000100:00100000:3.0:1789760743.416779:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732676947814 time 732710779648 state 0x0 ref 1 00000100:00100000:3.0:1789760743.416794:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732711040064 time 732710779648 state 0x1 ref 1 00000100:00100000:3.0:1789760743.416826:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732711040064 time 732747756891 state 0x0 ref 1 00000100:00100000:3.0:1789760743.416839:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484237056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.416889:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21491712, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.416898:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.416913:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760743.416917:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.416923:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21491712->21495807] (req 21491712->21495807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.416937:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21491712->21495807] (req 21491712->21495807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760743.450451:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972565 is committed 00002000:00010000:3.0:1789760743.450630:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21491712 -> 21495808 00010000:00010000:3.0:1789760743.450646:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21491712->21495807] (req 21491712->21495807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.450660:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21491712->21495807] (req 21491712->21495807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.450669:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21491712->21495807] (req 21491712->21495807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.450676:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.450681:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21491712->21495807] (req 21491712->21495807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.450689:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21491712->21495807] (req 21491712->21495807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.450709:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21491712->21495807] (req 21491712->21495807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.450715:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208936 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21491712->21495807] (req 21491712->21495807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.450845:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484237056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34004us (34332us total) trans 4294972565 rc 0/0 00000100:00100000:3.0:1789760743.450859:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5276 00000100:00100000:3.0:1789760743.450868:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732711040064 time 732747756891 state 0x2 ref 0 00000100:00100000:2.0:1789760743.455935:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760743.456026:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484237184 00000100:00100000:0.0:1789760743.456035:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484237184 00000100:00100000:0.0:1789760743.456137:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732711040064 time 732747756891 state 0x0 ref 1 00000100:00100000:0.0:1789760743.456153:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732748017307 time 732747756891 state 0x1 ref 1 00000100:00100000:0.0:1789760743.456181:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732748017307 time 732787111851 state 0x0 ref 1 00000100:00100000:0.0:1789760743.456195:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484237184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760743.456247:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21495808, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:0.0:1789760743.456255:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.456270:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:0.0:1789760743.456274:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760743.456281:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21495808->21499903] (req 21495808->21499903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.456297:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21495808->21499903] (req 21495808->21499903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760743.500209:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972566 is committed 00002000:00010000:3.0:1789760743.500363:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21495808 -> 21499904 00010000:00010000:3.0:1789760743.500378:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21495808->21499903] (req 21495808->21499903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.500391:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21495808->21499903] (req 21495808->21499903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.500400:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21495808->21499903] (req 21495808->21499903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.500408:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.500413:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21495808->21499903] (req 21495808->21499903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.500422:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21495808->21499903] (req 21495808->21499903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.500442:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21495808->21499903] (req 21495808->21499903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.500451:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620893d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21495808->21499903] (req 21495808->21499903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.500583:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484237184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44386us (44654us total) trans 4294972566 rc 0/0 00000100:00100000:3.0:1789760743.500594:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5277 00000100:00100000:3.0:1789760743.500605:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732748017307 time 732787111851 state 0x2 ref 0 00000100:00100000:0.0:1789760743.503626:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760743.503719:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484237312 00000100:00100000:2.0:1789760743.503729:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484237312 00000100:00100000:2.0:1789760743.503784:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732748017307 time 732787111851 state 0x0 ref 1 00000100:00100000:2.0:1789760743.503800:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732787372267 time 732787111851 state 0x1 ref 1 00000100:00100000:2.0:1789760743.503829:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732787372267 time 732834759519 state 0x0 ref 1 00000100:00100000:2.0:1789760743.503843:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484237312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760743.503899:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21499904, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:2.0:1789760743.503908:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.503924:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:2.0:1789760743.503928:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760743.503935:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21499904->21503999] (req 21499904->21503999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.503949:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21499904->21503999] (req 21499904->21503999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760743.541793:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972567 is committed 00002000:00010000:3.0:1789760743.541822:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21499904 -> 21504000 00010000:00010000:3.0:1789760743.541838:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21499904->21503999] (req 21499904->21503999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.541850:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21499904->21503999] (req 21499904->21503999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.541859:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21499904->21503999] (req 21499904->21503999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.541866:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.541871:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21499904->21503999] (req 21499904->21503999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.541879:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21499904->21503999] (req 21499904->21503999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.541898:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21499904->21503999] (req 21499904->21503999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.541911:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208944 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21499904->21503999] (req 21499904->21503999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.542002:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484237312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38157us (38381us total) trans 4294972567 rc 0/0 00000100:00100000:3.0:1789760743.542012:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5278 00000100:00100000:3.0:1789760743.542022:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732787372267 time 732834759519 state 0x2 ref 0 00000100:00100000:2.0:1789760743.545808:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.546291:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484237440 00000100:00100000:3.0:1789760743.546310:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484237440 00000100:00100000:3.0:1789760743.546368:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732787372267 time 732834759519 state 0x0 ref 1 00000100:00100000:3.0:1789760743.546383:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732835019935 time 732834759519 state 0x1 ref 1 00000100:00100000:3.0:1789760743.546508:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732835019935 time 732877437737 state 0x0 ref 1 00000100:00100000:3.0:1789760743.546521:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484237440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.546567:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21504000, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.546576:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.546590:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760743.546595:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.546600:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21504000->21508095] (req 21504000->21508095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.546619:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21504000->21508095] (req 21504000->21508095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760743.584019:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21504000 -> 21508096 00010000:00010000:3.0:1789760743.584040:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21504000->21508095] (req 21504000->21508095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.584053:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21504000->21508095] (req 21504000->21508095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.584062:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21504000->21508095] (req 21504000->21508095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.584107:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.584112:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21504000->21508095] (req 21504000->21508095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.584121:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21504000->21508095] (req 21504000->21508095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.584139:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21504000->21508095] (req 21504000->21508095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.584147:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620894b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21504000->21508095] (req 21504000->21508095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.584237:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484237440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37712us (38433us total) trans 4294972568 rc 0/0 00000001:00080000:1.0:1789760743.584249:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972568 is committed 00000100:00100000:3.0:1789760743.584249:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5279 00000100:00100000:3.0:1789760743.584262:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732835019935 time 732877437737 state 0x2 ref 0 00000100:00100000:3.0:1789760743.590619:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760743.590771:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484237568 00000100:00100000:2.0:1789760743.590780:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484237568 00000100:00100000:2.0:1789760743.590830:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732835019935 time 732877437737 state 0x0 ref 1 00000100:00100000:2.0:1789760743.590845:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732877698153 time 732877437737 state 0x1 ref 1 00000100:00100000:2.0:1789760743.590884:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732877698153 time 732921815066 state 0x0 ref 1 00000100:00100000:2.0:1789760743.590897:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484237568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760743.590951:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21508096, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:2.0:1789760743.590964:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.590980:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:2.0:1789760743.590985:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760743.590996:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21508096->21512191] (req 21508096->21512191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.591010:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21508096->21512191] (req 21508096->21512191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760743.636181:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972569 is committed 00002000:00010000:3.0:1789760743.636283:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21508096 -> 21512192 00010000:00010000:3.0:1789760743.636310:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21508096->21512191] (req 21508096->21512191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.636323:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21508096->21512191] (req 21508096->21512191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.636332:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21508096->21512191] (req 21508096->21512191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.636339:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.636344:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21508096->21512191] (req 21508096->21512191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.636353:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21508096->21512191] (req 21508096->21512191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.636371:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21508096->21512191] (req 21508096->21512191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.636383:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208952 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21508096->21512191] (req 21508096->21512191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.636512:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484237568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45613us (45898us total) trans 4294972569 rc 0/0 00000100:00100000:3.0:1789760743.636523:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5280 00000100:00100000:3.0:1789760743.636532:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732877698153 time 732921815066 state 0x2 ref 0 00000100:00100000:0.0:1789760743.642205:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.642335:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484237696 00000100:00100000:3.0:1789760743.642344:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484237696 00000100:00100000:3.0:1789760743.642397:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732877698153 time 732921815066 state 0x0 ref 1 00000100:00100000:3.0:1789760743.642412:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732922075482 time 732921815066 state 0x1 ref 1 00000100:00100000:3.0:1789760743.642443:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732922075482 time 732973373714 state 0x0 ref 1 00000100:00100000:3.0:1789760743.642456:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484237696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.642501:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21512192, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.642510:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.642526:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=732s v=(0 0) 00010000:00010000:3.0:1789760743.642530:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.642535:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21512192->21516287] (req 21512192->21516287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.642554:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21512192->21516287] (req 21512192->21516287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760743.674919:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972570 is committed 00002000:00010000:3.0:1789760743.675027:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21512192 -> 21516288 00010000:00010000:3.0:1789760743.675043:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21512192->21516287] (req 21512192->21516287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.675055:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21512192->21516287] (req 21512192->21516287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.675139:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21512192->21516287] (req 21512192->21516287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.675150:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.675159:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21512192->21516287] (req 21512192->21516287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.675168:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21512192->21516287] (req 21512192->21516287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.675192:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21512192->21516287] (req 21512192->21516287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.675200:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208959 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21512192->21516287] (req 21512192->21516287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.675376:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484237696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32916us (33171us total) trans 4294972570 rc 0/0 00000100:00100000:3.0:1789760743.675461:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5281 00000100:00100000:3.0:1789760743.675473:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732922075482 time 732973373714 state 0x2 ref 0 00000100:00100000:2.0:1789760743.679554:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760743.679777:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484237824 00000100:00100000:0.0:1789760743.679790:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484237824 00000100:00100000:0.0:1789760743.679861:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732922075482 time 732973373714 state 0x0 ref 1 00000100:00100000:0.0:1789760743.679877:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732973634130 time 732973373714 state 0x1 ref 1 00000100:00100000:0.0:1789760743.679910:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732973634130 time 733010839948 state 0x0 ref 1 00000100:00100000:0.0:1789760743.679925:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484237824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760743.679981:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21516288, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:0.0:1789760743.679992:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.680010:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:0.0:1789760743.680015:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760743.680022:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21516288->21520383] (req 21516288->21520383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.680037:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21516288->21520383] (req 21516288->21520383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760743.712109:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21516288 -> 21520384 00010000:00010000:3.0:1789760743.712130:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21516288->21520383] (req 21516288->21520383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.712144:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21516288->21520383] (req 21516288->21520383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.712152:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21516288->21520383] (req 21516288->21520383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.712159:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.712164:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21516288->21520383] (req 21516288->21520383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.712173:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21516288->21520383] (req 21516288->21520383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.712191:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21516288->21520383] (req 21516288->21520383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.712204:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208960 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21516288->21520383] (req 21516288->21520383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.712285:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484237824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32360us (32738us total) trans 4294972571 rc 0/0 00000100:00100000:3.0:1789760743.712296:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5282 00000100:00100000:3.0:1789760743.712315:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732973634130 time 733010839948 state 0x2 ref 0 00000001:00080000:1.0:1789760743.713090:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972571 is committed 00000100:00100000:0.0:1789760743.717975:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.718051:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484237952 00000100:00100000:3.0:1789760743.718060:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484237952 00000100:00100000:3.0:1789760743.718131:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 732973634130 time 733010839948 state 0x0 ref 1 00000100:00100000:3.0:1789760743.718146:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733011100364 time 733010839948 state 0x1 ref 1 00000100:00100000:3.0:1789760743.718171:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733011100364 time 733049102668 state 0x0 ref 1 00000100:00100000:3.0:1789760743.718189:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484237952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.718231:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21520384, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.718239:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.718255:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:3.0:1789760743.718259:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.718265:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21520384->21524479] (req 21520384->21524479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.718284:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21520384->21524479] (req 21520384->21524479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760743.763785:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21520384 -> 21524480 00010000:00010000:3.0:1789760743.763804:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21520384->21524479] (req 21520384->21524479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.763817:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21520384->21524479] (req 21520384->21524479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.763825:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21520384->21524479] (req 21520384->21524479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.763832:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.763836:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21520384->21524479] (req 21520384->21524479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.763845:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21520384->21524479] (req 21520384->21524479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.763862:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21520384->21524479] (req 21520384->21524479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.763870:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208967 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21520384->21524479] (req 21520384->21524479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760743.764174:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972572 is committed 00000100:00100000:3.0:1789760743.764276:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484237952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46082us (46305us total) trans 4294972572 rc 0/0 00000100:00100000:3.0:1789760743.764286:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5283 00000100:00100000:3.0:1789760743.764309:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733011100364 time 733049102668 state 0x2 ref 0 00000100:00100000:0.0:1789760743.771194:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.771553:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484238080 00000100:00100000:3.0:1789760743.771563:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484238080 00000100:00100000:3.0:1789760743.771623:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733011100364 time 733049102668 state 0x0 ref 1 00000100:00100000:3.0:1789760743.771637:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733049363084 time 733049102668 state 0x1 ref 1 00000100:00100000:3.0:1789760743.771859:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733049363084 time 733102788206 state 0x0 ref 1 00000100:00100000:3.0:1789760743.771873:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484238080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.771922:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21524480, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.771930:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.771946:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:3.0:1789760743.771950:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.771956:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21524480->21528575] (req 21524480->21528575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.771982:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21524480->21528575] (req 21524480->21528575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760743.801190:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972573 is committed 00002000:00010000:3.0:1789760743.801441:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21524480 -> 21528576 00010000:00010000:3.0:1789760743.801457:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21524480->21528575] (req 21524480->21528575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.801469:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21524480->21528575] (req 21524480->21528575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.801479:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21524480->21528575] (req 21524480->21528575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.801486:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.801491:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21524480->21528575] (req 21524480->21528575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.801499:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21524480->21528575] (req 21524480->21528575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.801520:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21524480->21528575] (req 21524480->21528575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.801528:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620896e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21524480->21528575] (req 21524480->21528575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.801734:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484238080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29857us (30546us total) trans 4294972573 rc 0/0 00000100:00100000:3.0:1789760743.801748:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5284 00000100:00100000:3.0:1789760743.801758:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733049363084 time 733102788206 state 0x2 ref 0 00000100:00100000:2.0:1789760743.805570:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760743.805743:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484238208 00000100:00100000:0.0:1789760743.805753:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484238208 00000100:00100000:0.0:1789760743.805812:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733049363084 time 733102788206 state 0x0 ref 1 00000100:00100000:0.0:1789760743.805828:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733103048622 time 733102788206 state 0x1 ref 1 00000100:00100000:0.0:1789760743.805860:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733103048622 time 733136791237 state 0x0 ref 1 00000100:00100000:0.0:1789760743.805875:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484238208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760743.805926:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21528576, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:0.0:1789760743.805934:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.805950:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:0.0:1789760743.805954:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760743.805960:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21528576->21532671] (req 21528576->21532671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.805973:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21528576->21532671] (req 21528576->21532671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760743.820491:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9471880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508922368:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760743.820558:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760743.820622:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508922368 00000100:00100000:1.0:1789760743.820629:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508922368 00000100:00100000:1.0:1789760743.820695:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 442 00000100:00100000:1.0:1789760743.820705:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508922368:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760743.820741:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760743.820786:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9472a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508922368:12345-0@lo:41:osp-pre-1-0.0 Request processed in 82us (228us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760743.820793:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 442 00000100:00100000:1.0:1789760743.820837:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9471880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508922368:0@lo:41:osp-pre-1-0.0 00002000:00010000:3.0:1789760743.842233:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21528576 -> 21532672 00010000:00010000:3.0:1789760743.842253:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21528576->21532671] (req 21528576->21532671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.842266:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21528576->21532671] (req 21528576->21532671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.842276:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21528576->21532671] (req 21528576->21532671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.842284:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.842307:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21528576->21532671] (req 21528576->21532671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.842323:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21528576->21532671] (req 21528576->21532671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.842347:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21528576->21532671] (req 21528576->21532671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.842355:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208975 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21528576->21532671] (req 21528576->21532671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.842456:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484238208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36580us (36890us total) trans 4294972574 rc 0/0 00000100:00100000:3.0:1789760743.842467:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5285 00000100:00100000:3.0:1789760743.842476:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733103048622 time 733136791237 state 0x2 ref 0 00000001:00080000:1.0:1789760743.843260:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972574 is committed 00000100:00100000:0.0:1789760743.847276:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760743.847498:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484238336 00000100:00100000:3.0:1789760743.847509:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484238336 00000100:00100000:3.0:1789760743.847565:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733103048622 time 733136791237 state 0x0 ref 1 00000100:00100000:3.0:1789760743.847582:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733137051653 time 733136791237 state 0x1 ref 1 00000100:00100000:3.0:1789760743.847618:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733137051653 time 733178548708 state 0x0 ref 1 00000100:00100000:3.0:1789760743.847631:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484238336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760743.847682:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21532672, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760743.847692:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.847707:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:3.0:1789760743.847712:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760743.847718:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21532672->21536767] (req 21532672->21536767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.847733:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21532672->21536767] (req 21532672->21536767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760743.906956:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972575 is committed 00002000:00010000:3.0:1789760743.907269:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21532672 -> 21536768 00010000:00010000:3.0:1789760743.907367:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21532672->21536767] (req 21532672->21536767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.907382:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21532672->21536767] (req 21532672->21536767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.907392:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21532672->21536767] (req 21532672->21536767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.907400:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760743.907406:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21532672->21536767] (req 21532672->21536767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.907416:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21532672->21536767] (req 21532672->21536767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.907442:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21532672->21536767] (req 21532672->21536767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760743.907450:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620897c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21532672->21536767] (req 21532672->21536767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760743.908022:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484238336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60385us (60747us total) trans 4294972575 rc 0/0 00000100:00100000:3.0:1789760743.908040:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5286 00000100:00100000:3.0:1789760743.908053:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733137051653 time 733178548708 state 0x2 ref 0 00000100:00100000:0.0:1789760743.914915:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760743.915035:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484238464 00000100:00100000:2.0:1789760743.915048:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484238464 00000100:00100000:2.0:1789760743.915144:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733137051653 time 733178548708 state 0x0 ref 1 00000100:00100000:2.0:1789760743.915163:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733178809124 time 733178548708 state 0x1 ref 1 00000100:00100000:2.0:1789760743.915199:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733178809124 time 733246129678 state 0x0 ref 1 00000100:00100000:2.0:1789760743.915215:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484238464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760743.915277:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21536768, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:2.0:1789760743.915288:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.915319:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:2.0:1789760743.915325:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760743.915332:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21536768->21540863] (req 21536768->21540863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760743.915349:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21536768->21540863] (req 21536768->21540863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760743.992035:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21536768 -> 21540864 00010000:00010000:0.0:1789760743.992082:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21536768->21540863] (req 21536768->21540863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.992099:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21536768->21540863] (req 21536768->21540863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.992111:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21536768->21540863] (req 21536768->21540863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.992119:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760743.992124:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21536768->21540863] (req 21536768->21540863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.992134:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21536768->21540863] (req 21536768->21540863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.992159:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21536768->21540863] (req 21536768->21540863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760743.992168:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208983 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21536768->21540863] (req 21536768->21540863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760743.992308:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484238464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77079us (77390us total) trans 4294972576 rc 0/0 00000100:00100000:0.0:1789760743.992320:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5287 00000100:00100000:0.0:1789760743.992333:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733178809124 time 733246129678 state 0x2 ref 0 00000001:00080000:1.0:1789760743.999580:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972576 is committed 00000100:00100000:2.0:1789760743.999670:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760744.003404:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484238592 00000100:00100000:3.0:1789760744.003426:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484238592 00000100:00100000:3.0:1789760744.003535:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733178809124 time 733246129678 state 0x0 ref 1 00000100:00100000:3.0:1789760744.003555:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733246390094 time 733246129678 state 0x1 ref 1 00000100:00100000:3.0:1789760744.003595:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733246390094 time 733334525847 state 0x0 ref 1 00000100:00100000:3.0:1789760744.003612:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484238592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760744.003684:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21540864, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760744.003697:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.003718:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:3.0:1789760744.003724:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760744.003731:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21540864->21544959] (req 21540864->21544959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.003749:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21540864->21544959] (req 21540864->21544959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1789760744.004411:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:1.0:1789760744.041898:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972577 is committed 00002000:00010000:0.0:1789760744.042168:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21540864 -> 21544960 00010000:00010000:0.0:1789760744.042191:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21540864->21544959] (req 21540864->21544959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.042416:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21540864->21544959] (req 21540864->21544959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.042428:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21540864->21544959] (req 21540864->21544959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.042437:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760744.042442:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21540864->21544959] (req 21540864->21544959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.042453:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21540864->21544959] (req 21540864->21544959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.042482:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21540864->21544959] (req 21540864->21544959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.042490:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620898a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21540864->21544959] (req 21540864->21544959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760744.042672:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484238592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39056us (43000us total) trans 4294972577 rc 0/0 00000100:00100000:0.0:1789760744.042689:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5288 00000100:00100000:0.0:1789760744.042701:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733246390094 time 733334525847 state 0x2 ref 0 00000100:00100000:0.0:1789760744.052251:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760744.052373:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484238720 00000100:00100000:3.0:1789760744.052387:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484238720 00000100:00100000:3.0:1789760744.052476:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733246390094 time 733334525847 state 0x0 ref 1 00000100:00100000:3.0:1789760744.052495:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733334786263 time 733334525847 state 0x1 ref 1 00000100:00100000:3.0:1789760744.052636:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733334786263 time 733383564997 state 0x0 ref 1 00000100:00100000:3.0:1789760744.052653:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484238720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760744.052720:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21544960, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:3.0:1789760744.052731:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.052750:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:3.0:1789760744.052755:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760744.052763:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21544960->21549055] (req 21544960->21549055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.052779:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21544960->21549055] (req 21544960->21549055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760744.084890:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972578 is committed 00002000:00010000:0.0:1789760744.085024:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21544960 -> 21549056 00010000:00010000:0.0:1789760744.085045:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21544960->21549055] (req 21544960->21549055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.085060:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21544960->21549055] (req 21544960->21549055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.085106:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21544960->21549055] (req 21544960->21549055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.085115:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760744.085121:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21544960->21549055] (req 21544960->21549055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.085131:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21544960->21549055] (req 21544960->21549055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.085158:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21544960->21549055] (req 21544960->21549055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.085167:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208991 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21544960->21549055] (req 21544960->21549055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760744.085304:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484238720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32640us (33055us total) trans 4294972578 rc 0/0 00000100:00100000:0.0:1789760744.085316:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5289 00000100:00100000:0.0:1789760744.085329:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733334786263 time 733383564997 state 0x2 ref 0 00000100:00100000:2.0:1789760744.091344:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760744.091484:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484238848 00000100:00100000:0.0:1789760744.091498:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484238848 00000100:00100000:0.0:1789760744.091577:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733334786263 time 733383564997 state 0x0 ref 1 00000100:00100000:0.0:1789760744.091596:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733383825413 time 733383564997 state 0x1 ref 1 00000100:00100000:0.0:1789760744.091677:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733383825413 time 733422607221 state 0x0 ref 1 00000100:00100000:0.0:1789760744.091695:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484238848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760744.091763:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21549056, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:0.0:1789760744.091774:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.091806:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:0.0:1789760744.091812:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760744.091819:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21549056->21553151] (req 21549056->21553151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.091837:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21549056->21553151] (req 21549056->21553151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760744.127639:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972579 is committed 00002000:00010000:3.0:1789760744.127995:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21549056 -> 21553152 00010000:00010000:3.0:1789760744.128014:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21549056->21553151] (req 21549056->21553151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.128029:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21549056->21553151] (req 21549056->21553151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.128039:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21549056->21553151] (req 21549056->21553151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.128048:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.128053:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21549056->21553151] (req 21549056->21553151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.128062:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21549056->21553151] (req 21549056->21553151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.128179:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21549056->21553151] (req 21549056->21553151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.128186:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208998 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21549056->21553151] (req 21549056->21553151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760744.128326:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484238848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36627us (36989us total) trans 4294972579 rc 0/0 00000100:00100000:3.0:1789760744.128341:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5290 00000100:00100000:3.0:1789760744.128357:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733383825413 time 733422607221 state 0x2 ref 0 00000100:00100000:2.0:1789760744.131309:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760744.131405:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484238976 00000100:00100000:0.0:1789760744.131419:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484238976 00000100:00100000:0.0:1789760744.131499:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733383825413 time 733422607221 state 0x0 ref 1 00000100:00100000:0.0:1789760744.131518:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733422867637 time 733422607221 state 0x1 ref 1 00000100:00100000:0.0:1789760744.131555:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733422867637 time 733462485185 state 0x0 ref 1 00000100:00100000:0.0:1789760744.131569:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484238976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760744.131637:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21553152, mtime: 0x6aad94e7, atime: 0x0, ctime: 0x6aad94e7, blocks: 0xa807 00010000:00010000:0.0:1789760744.131647:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.131668:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:0.0:1789760744.131673:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760744.131681:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21553152->21557247] (req 21553152->21557247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.131698:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21553152->21557247] (req 21553152->21557247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760744.169748:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21553152 -> 21557248 00002000:00010000:0.0:1789760744.169762:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760743 -> 1789760744 00002000:00010000:0.0:1789760744.169765:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760743 -> 1789760744 00010000:00010000:0.0:1789760744.169782:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21553152->21557247] (req 21553152->21557247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.169798:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21553152->21557247] (req 21553152->21557247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.169810:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21553152->21557247] (req 21553152->21557247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.169818:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760744.169824:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21553152->21557247] (req 21553152->21557247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.169835:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21553152->21557247] (req 21553152->21557247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760744.169844:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972580 is committed 00010000:00010000:0.0:1789760744.169859:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21553152->21557247] (req 21553152->21557247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.169868:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620899f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21553152->21557247] (req 21553152->21557247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760744.170059:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484238976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38484us (38768us total) trans 4294972580 rc 0/0 00000100:00100000:0.0:1789760744.170360:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5291 00000100:00100000:0.0:1789760744.170373:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733422867637 time 733462485185 state 0x2 ref 0 00000100:00100000:0.0:1789760744.173630:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760744.173745:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484239104 00000100:00100000:3.0:1789760744.173758:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484239104 00000100:00100000:3.0:1789760744.173830:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733422867637 time 733462485185 state 0x0 ref 1 00000100:00100000:3.0:1789760744.173849:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733462745601 time 733462485185 state 0x1 ref 1 00000100:00100000:3.0:1789760744.173914:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733462745601 time 733504843100 state 0x0 ref 1 00000100:00100000:3.0:1789760744.173934:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484239104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760744.174011:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21557248, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:3.0:1789760744.174022:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.174043:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:3.0:1789760744.174048:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760744.174056:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21557248->21561343] (req 21557248->21561343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.174110:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21557248->21561343] (req 21557248->21561343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760744.202220:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21557248 -> 21561344 00010000:00010000:3.0:1789760744.202246:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21557248->21561343] (req 21557248->21561343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.202261:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21557248->21561343] (req 21557248->21561343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.202271:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21557248->21561343] (req 21557248->21561343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.202279:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.202285:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21557248->21561343] (req 21557248->21561343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.202737:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21557248->21561343] (req 21557248->21561343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.202781:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21557248->21561343] (req 21557248->21561343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.202792:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089a6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21557248->21561343] (req 21557248->21561343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760744.202919:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972581 is committed 00000100:00100000:3.0:1789760744.202930:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484239104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28995us (29311us total) trans 4294972581 rc 0/0 00000100:00100000:3.0:1789760744.202943:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5292 00000100:00100000:3.0:1789760744.202956:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733462745601 time 733504843100 state 0x2 ref 0 00000100:00100000:3.0:1789760744.212409:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760744.212689:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484239232 00000100:00100000:0.0:1789760744.212701:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484239232 00000100:00100000:0.0:1789760744.212782:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733462745601 time 733504843100 state 0x0 ref 1 00000100:00100000:0.0:1789760744.212799:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733505103516 time 733504843100 state 0x1 ref 1 00000100:00100000:0.0:1789760744.212836:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733505103516 time 733543766772 state 0x0 ref 1 00000100:00100000:0.0:1789760744.212851:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484239232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760744.212934:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21561344, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:0.0:1789760744.212946:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.212967:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:0.0:1789760744.212972:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760744.212981:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21561344->21565439] (req 21561344->21565439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.212998:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21561344->21565439] (req 21561344->21565439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760744.252410:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972582 is committed 00002000:00010000:3.0:1789760744.252559:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21561344 -> 21565440 00010000:00010000:3.0:1789760744.252574:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21561344->21565439] (req 21561344->21565439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.252587:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21561344->21565439] (req 21561344->21565439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.252597:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21561344->21565439] (req 21561344->21565439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.252611:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.252616:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21561344->21565439] (req 21561344->21565439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.252626:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21561344->21565439] (req 21561344->21565439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.252763:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21561344->21565439] (req 21561344->21565439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.252774:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962089ad lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21561344->21565439] (req 21561344->21565439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760744.252910:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484239232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40058us (40506us total) trans 4294972582 rc 0/0 00000100:00100000:3.0:1789760744.252921:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5293 00000100:00100000:3.0:1789760744.252932:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733505103516 time 733543766772 state 0x2 ref 0 00000100:00100000:3.0:1789760744.257139:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760744.257266:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484239360 00000100:00100000:0.0:1789760744.257278:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484239360 00000100:00100000:0.0:1789760744.257546:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733505103516 time 733543766772 state 0x0 ref 1 00000100:00100000:0.0:1789760744.257561:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733544027188 time 733543766772 state 0x1 ref 1 00000100:00100000:0.0:1789760744.257621:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733544027188 time 733588552145 state 0x0 ref 1 00000100:00100000:0.0:1789760744.257635:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484239360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760744.257684:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21565440, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:0.0:1789760744.257693:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.257710:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:0.0:1789760744.257715:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760744.257720:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21565440->21569535] (req 21565440->21569535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.257735:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21565440->21569535] (req 21565440->21569535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760744.291576:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972583 is committed 00002000:00010000:3.0:1789760744.291749:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21565440 -> 21569536 00010000:00010000:3.0:1789760744.291764:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21565440->21569535] (req 21565440->21569535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.291777:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21565440->21569535] (req 21565440->21569535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.291791:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21565440->21569535] (req 21565440->21569535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.291798:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.291803:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21565440->21569535] (req 21565440->21569535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.291812:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21565440->21569535] (req 21565440->21569535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.291832:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21565440->21569535] (req 21565440->21569535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.291840:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962089b4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21565440->21569535] (req 21565440->21569535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760744.292033:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484239360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34395us (34899us total) trans 4294972583 rc 0/0 00000100:00100000:3.0:1789760744.292043:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5294 00000100:00100000:3.0:1789760744.292052:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733544027188 time 733588552145 state 0x2 ref 0 00000100:00100000:1.0:1789760744.299836:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760744.300018:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484239488 00000100:00100000:2.0:1789760744.300027:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484239488 00000100:00100000:2.0:1789760744.300133:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733544027188 time 733588552145 state 0x0 ref 1 00000100:00100000:2.0:1789760744.300147:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733588812561 time 733588552145 state 0x1 ref 1 00000100:00100000:2.0:1789760744.300216:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733588812561 time 733631146383 state 0x0 ref 1 00000100:00100000:2.0:1789760744.300228:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484239488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760744.300275:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21569536, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:2.0:1789760744.300283:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.300309:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:2.0:1789760744.300314:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760744.300320:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21569536->21573631] (req 21569536->21573631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.300334:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21569536->21573631] (req 21569536->21573631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760744.326682:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972584 is committed 00002000:00010000:2.0:1789760744.326849:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21569536 -> 21573632 00010000:00010000:2.0:1789760744.326864:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21569536->21573631] (req 21569536->21573631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.326878:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21569536->21573631] (req 21569536->21573631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.326886:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21569536->21573631] (req 21569536->21573631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.327047:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760744.327052:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21569536->21573631] (req 21569536->21573631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.327147:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21569536->21573631] (req 21569536->21573631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.327171:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21569536->21573631] (req 21569536->21573631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.327180:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089bb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21569536->21573631] (req 21569536->21573631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760744.327281:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484239488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27049us (27447us total) trans 4294972584 rc 0/0 00000100:00100000:2.0:1789760744.327291:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5295 00000100:00100000:2.0:1789760744.327310:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733588812561 time 733631146383 state 0x2 ref 0 00010000:00010000:3.0:1789760744.332381:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760744.333600:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760744.333684:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484239616 00000100:00100000:3.0:1789760744.333694:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484239616 00000100:00100000:3.0:1789760744.333744:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733588812561 time 733631146383 state 0x0 ref 1 00000100:00100000:3.0:1789760744.333766:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733631406799 time 733631146383 state 0x1 ref 1 00000100:00100000:3.0:1789760744.333830:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733631406799 time 733664761075 state 0x0 ref 1 00000100:00100000:3.0:1789760744.333843:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484239616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760744.333888:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21573632, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:3.0:1789760744.333907:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.333922:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:3.0:1789760744.333926:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760744.333933:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21573632->21577727] (req 21573632->21577727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.333946:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21573632->21577727] (req 21573632->21577727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760744.360664:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972585 is committed 00002000:00010000:3.0:1789760744.360665:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21573632 -> 21577728 00010000:00010000:3.0:1789760744.360680:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21573632->21577727] (req 21573632->21577727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.360693:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21573632->21577727] (req 21573632->21577727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.360706:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21573632->21577727] (req 21573632->21577727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.360720:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.360725:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21573632->21577727] (req 21573632->21577727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.360735:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21573632->21577727] (req 21573632->21577727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.360755:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21573632->21577727] (req 21573632->21577727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.360763:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089c2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21573632->21577727] (req 21573632->21577727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760744.360850:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484239616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27004us (27252us total) trans 4294972585 rc 0/0 00000100:00100000:3.0:1789760744.360860:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5296 00000100:00100000:3.0:1789760744.360868:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733631406799 time 733664761075 state 0x2 ref 0 00000100:00100000:3.0:1789760744.363635:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760744.363914:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484239744 00000100:00100000:0.0:1789760744.363922:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484239744 00000100:00100000:0.0:1789760744.363971:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733631406799 time 733664761075 state 0x0 ref 1 00000100:00100000:0.0:1789760744.363983:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733665021491 time 733664761075 state 0x1 ref 1 00000100:00100000:0.0:1789760744.364016:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733665021491 time 733694946725 state 0x0 ref 1 00000100:00100000:0.0:1789760744.364045:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484239744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760744.364143:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21577728, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:0.0:1789760744.364160:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.364175:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:0.0:1789760744.364179:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760744.364191:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21577728->21581823] (req 21577728->21581823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.364636:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21577728->21581823] (req 21577728->21581823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760744.407829:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972586 is committed 00002000:00010000:3.0:1789760744.407953:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21577728 -> 21581824 00010000:00010000:3.0:1789760744.407969:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21577728->21581823] (req 21577728->21581823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.407983:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21577728->21581823] (req 21577728->21581823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.407999:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21577728->21581823] (req 21577728->21581823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.408007:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.408013:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21577728->21581823] (req 21577728->21581823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.408023:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21577728->21581823] (req 21577728->21581823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.408043:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21577728->21581823] (req 21577728->21581823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.408057:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089c9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21577728->21581823] (req 21577728->21581823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760744.408197:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484239744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44149us (44566us total) trans 4294972586 rc 0/0 00000100:00100000:3.0:1789760744.408208:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5297 00000100:00100000:3.0:1789760744.408218:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733665021491 time 733694946725 state 0x2 ref 0 00000100:00100000:1.0:1789760744.414232:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760744.414348:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484239872 00000100:00100000:0.0:1789760744.414358:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484239872 00000100:00100000:0.0:1789760744.414418:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733665021491 time 733694946725 state 0x0 ref 1 00000100:00100000:0.0:1789760744.414433:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733695207141 time 733694946725 state 0x1 ref 1 00000100:00100000:0.0:1789760744.414504:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733695207141 time 733745433764 state 0x0 ref 1 00000100:00100000:0.0:1789760744.414520:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484239872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760744.414573:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21581824, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:0.0:1789760744.414582:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.414597:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:0.0:1789760744.414604:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760744.414611:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21581824->21585919] (req 21581824->21585919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.414626:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21581824->21585919] (req 21581824->21585919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760744.445091:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972587 is committed 00002000:00010000:0.0:1789760744.445531:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21581824 -> 21585920 00010000:00010000:0.0:1789760744.445549:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21581824->21585919] (req 21581824->21585919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.445574:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21581824->21585919] (req 21581824->21585919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.445584:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21581824->21585919] (req 21581824->21585919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.445591:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760744.445596:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21581824->21585919] (req 21581824->21585919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.445605:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21581824->21585919] (req 21581824->21585919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.445625:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21581824->21585919] (req 21581824->21585919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.445637:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089d0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21581824->21585919] (req 21581824->21585919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760744.445790:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484239872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31269us (31563us total) trans 4294972587 rc 0/0 00000100:00100000:0.0:1789760744.445801:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5298 00000100:00100000:0.0:1789760744.445810:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733695207141 time 733745433764 state 0x2 ref 0 00000100:00100000:1.0:1789760744.449536:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760744.451024:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484240000 00000100:00100000:2.0:1789760744.451036:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484240000 00000100:00100000:2.0:1789760744.451317:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733695207141 time 733745433764 state 0x0 ref 1 00000100:00100000:2.0:1789760744.451391:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733745694180 time 733745433764 state 0x1 ref 1 00000100:00100000:2.0:1789760744.451462:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733745694180 time 733782392328 state 0x0 ref 1 00000100:00100000:2.0:1789760744.451476:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484240000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760744.451525:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21585920, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:2.0:1789760744.451534:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.451554:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:2.0:1789760744.451558:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760744.451565:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21585920->21590015] (req 21585920->21590015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.451578:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21585920->21590015] (req 21585920->21590015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760744.501838:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21585920 -> 21590016 00010000:00010000:2.0:1789760744.501857:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21585920->21590015] (req 21585920->21590015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.501869:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21585920->21590015] (req 21585920->21590015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.501878:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21585920->21590015] (req 21585920->21590015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.501886:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760744.501890:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21585920->21590015] (req 21585920->21590015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.501901:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21585920->21590015] (req 21585920->21590015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.501921:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21585920->21590015] (req 21585920->21590015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.501931:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962089d7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21585920->21590015] (req 21585920->21590015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760744.502021:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484240000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50542us (52489us total) trans 4294972588 rc 0/0 00000100:00100000:2.0:1789760744.502032:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5299 00000100:00100000:2.0:1789760744.502042:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733745694180 time 733782392328 state 0x2 ref 0 00000001:00080000:2.0:1789760744.502120:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972588 is committed 00000100:00100000:3.0:1789760744.505343:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760744.508215:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484240128 00000100:00100000:0.0:1789760744.508231:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484240128 00000100:00100000:1.0:1789760744.511876:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733745694180 time 733782392328 state 0x0 ref 1 00000100:00100000:1.0:1789760744.511893:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733782652744 time 733782392328 state 0x1 ref 1 00000100:00100000:1.0:1789760744.511931:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733782652744 time 733842862339 state 0x0 ref 1 00000100:00100000:1.0:1789760744.511944:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484240128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760744.511990:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21590016, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:1.0:1789760744.511998:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.512012:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:1.0:1789760744.512017:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760744.512023:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21590016->21594111] (req 21590016->21594111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.512036:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21590016->21594111] (req 21590016->21594111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760744.552693:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972589 is committed 00002000:00010000:1.0:1789760744.552795:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21590016 -> 21594112 00010000:00010000:1.0:1789760744.552810:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21590016->21594111] (req 21590016->21594111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.552822:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21590016->21594111] (req 21590016->21594111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.552831:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21590016->21594111] (req 21590016->21594111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.552838:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760744.552843:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21590016->21594111] (req 21590016->21594111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.552852:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21590016->21594111] (req 21590016->21594111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.552871:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21590016->21594111] (req 21590016->21594111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.552879:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962089de lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21590016->21594111] (req 21590016->21594111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760744.552982:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484240128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41035us (47643us total) trans 4294972589 rc 0/0 00000100:00100000:1.0:1789760744.552992:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5300 00000100:00100000:1.0:1789760744.553001:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733782652744 time 733842862339 state 0x2 ref 0 00000100:00100000:0.0:1789760744.556608:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760744.556716:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484240256 00000100:00100000:3.0:1789760744.556726:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484240256 00000100:00100000:3.0:1789760744.556783:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733782652744 time 733842862339 state 0x0 ref 1 00000100:00100000:3.0:1789760744.556802:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733843122755 time 733842862339 state 0x1 ref 1 00000100:00100000:3.0:1789760744.556884:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733843122755 time 733887813467 state 0x0 ref 1 00000100:00100000:3.0:1789760744.556898:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484240256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760744.556950:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21594112, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:3.0:1789760744.556959:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.556975:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:3.0:1789760744.556980:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760744.556987:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21594112->21598207] (req 21594112->21598207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.557001:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21594112->21598207] (req 21594112->21598207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760744.587556:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972590 is committed 00002000:00010000:3.0:1789760744.587733:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21594112 -> 21598208 00010000:00010000:3.0:1789760744.587751:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21594112->21598207] (req 21594112->21598207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.587763:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21594112->21598207] (req 21594112->21598207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.587771:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21594112->21598207] (req 21594112->21598207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.587783:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.587788:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21594112->21598207] (req 21594112->21598207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.587796:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21594112->21598207] (req 21594112->21598207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.587817:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21594112->21598207] (req 21594112->21598207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.587824:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962089e5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21594112->21598207] (req 21594112->21598207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760744.587921:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484240256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31020us (31318us total) trans 4294972590 rc 0/0 00000100:00100000:3.0:1789760744.587930:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5301 00000100:00100000:3.0:1789760744.587940:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733843122755 time 733887813467 state 0x2 ref 0 00000100:00100000:2.0:1789760744.592421:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760744.592765:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484240384 00000100:00100000:3.0:1789760744.592775:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484240384 00000100:00100000:3.0:1789760744.592831:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733843122755 time 733887813467 state 0x0 ref 1 00000100:00100000:3.0:1789760744.592847:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733888073883 time 733887813467 state 0x1 ref 1 00000100:00100000:3.0:1789760744.592880:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733888073883 time 733923810280 state 0x0 ref 1 00000100:00100000:3.0:1789760744.592896:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484240384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760744.592947:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21598208, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:3.0:1789760744.592956:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.592972:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:3.0:1789760744.592976:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760744.592983:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21598208->21602303] (req 21598208->21602303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.592998:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21598208->21602303] (req 21598208->21602303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760744.631764:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21598208 -> 21602304 00010000:00010000:1.0:1789760744.631785:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21598208->21602303] (req 21598208->21602303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.631797:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21598208->21602303] (req 21598208->21602303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.631806:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21598208->21602303] (req 21598208->21602303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.631814:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760744.631818:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21598208->21602303] (req 21598208->21602303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.631827:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21598208->21602303] (req 21598208->21602303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.631846:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21598208->21602303] (req 21598208->21602303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.631859:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962089ec lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21598208->21602303] (req 21598208->21602303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760744.631957:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484240384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39058us (39542us total) trans 4294972591 rc 0/0 00000100:00100000:1.0:1789760744.631967:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5302 00000100:00100000:1.0:1789760744.631977:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733888073883 time 733923810280 state 0x2 ref 0 00000001:00080000:2.0:1789760744.632060:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972591 is committed 00000100:00100000:1.0:1789760744.637372:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760744.637580:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484240512 00000100:00100000:0.0:1789760744.637590:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484240512 00000100:00100000:0.0:1789760744.637649:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733888073883 time 733923810280 state 0x0 ref 1 00000100:00100000:0.0:1789760744.637665:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733924070696 time 733923810280 state 0x1 ref 1 00000100:00100000:0.0:1789760744.637733:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733924070696 time 733968662509 state 0x0 ref 1 00000100:00100000:0.0:1789760744.637750:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484240512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760744.637804:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21602304, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:0.0:1789760744.637813:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.637829:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=733s v=(0 0) 00010000:00010000:0.0:1789760744.637834:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760744.637840:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21602304->21606399] (req 21602304->21606399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.637853:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21602304->21606399] (req 21602304->21606399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760744.694411:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21602304 -> 21606400 00010000:00010000:2.0:1789760744.694430:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21602304->21606399] (req 21602304->21606399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.694444:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21602304->21606399] (req 21602304->21606399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.694453:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21602304->21606399] (req 21602304->21606399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.694462:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760744.694467:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21602304->21606399] (req 21602304->21606399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.694482:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21602304->21606399] (req 21602304->21606399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.694504:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21602304->21606399] (req 21602304->21606399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.694511:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962089f3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21602304->21606399] (req 21602304->21606399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760744.694614:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484240512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56864us (57258us total) trans 4294972592 rc 0/0 00000100:00100000:2.0:1789760744.694625:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5303 00000100:00100000:2.0:1789760744.694634:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733924070696 time 733968662509 state 0x2 ref 0 00000001:00080000:1.0:1789760744.695465:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972592 is committed 00000100:00100000:3.0:1789760744.697815:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760744.697947:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484240640 00000100:00100000:1.0:1789760744.697961:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484240640 00000100:00100000:1.0:1789760744.698040:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733924070696 time 733968662509 state 0x0 ref 1 00000100:00100000:1.0:1789760744.698059:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733968922925 time 733968662509 state 0x1 ref 1 00000100:00100000:1.0:1789760744.698153:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733968922925 time 734029082267 state 0x0 ref 1 00000100:00100000:1.0:1789760744.698167:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484240640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760744.698244:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21606400, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:1.0:1789760744.698257:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.698280:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:1.0:1789760744.698285:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760744.698302:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21606400->21610495] (req 21606400->21610495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.698320:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21606400->21610495] (req 21606400->21610495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760744.749888:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21606400 -> 21610496 00010000:00010000:2.0:1789760744.749915:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21606400->21610495] (req 21606400->21610495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.749931:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21606400->21610495] (req 21606400->21610495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.749942:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21606400->21610495] (req 21606400->21610495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.749950:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760744.749956:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21606400->21610495] (req 21606400->21610495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.749968:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21606400->21610495] (req 21606400->21610495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.749992:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21606400->21610495] (req 21606400->21610495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.750000:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962089fa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21606400->21610495] (req 21606400->21610495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760744.750171:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484240640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51997us (52364us total) trans 4294972593 rc 0/0 00000100:00100000:2.0:1789760744.750185:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5304 00000100:00100000:2.0:1789760744.750198:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733968922925 time 734029082267 state 0x2 ref 0 00000001:00080000:1.0:1789760744.750799:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972593 is committed 00000100:00100000:3.0:1789760744.752849:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760744.752962:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484240768 00000100:00100000:2.0:1789760744.752975:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484240768 00000100:00100000:2.0:1789760744.753055:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 733968922925 time 734029082267 state 0x0 ref 1 00000100:00100000:2.0:1789760744.753120:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734029342683 time 734029082267 state 0x1 ref 1 00000100:00100000:2.0:1789760744.753149:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734029342683 time 734084079909 state 0x0 ref 1 00000100:00100000:2.0:1789760744.753165:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484240768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760744.753235:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21610496, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:2.0:1789760744.753247:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.753269:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:2.0:1789760744.753275:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760744.753283:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21610496->21614591] (req 21610496->21614591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.753859:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21610496->21614591] (req 21610496->21614591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760744.807225:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21610496 -> 21614592 00010000:00010000:3.0:1789760744.807255:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21610496->21614591] (req 21610496->21614591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.807272:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21610496->21614591] (req 21610496->21614591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.807283:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21610496->21614591] (req 21610496->21614591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.807303:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.807309:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21610496->21614591] (req 21610496->21614591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.807321:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21610496->21614591] (req 21610496->21614591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.807346:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21610496->21614591] (req 21610496->21614591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.807354:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208a01 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21610496->21614591] (req 21610496->21614591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760744.807698:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972594 is committed 00000100:00100000:3.0:1789760744.807760:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484240768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54588us (54914us total) trans 4294972594 rc 0/0 00000100:00100000:3.0:1789760744.807773:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5305 00000100:00100000:3.0:1789760744.807785:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734029342683 time 734084079909 state 0x2 ref 0 00000100:00100000:0.0:1789760744.810976:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760744.811658:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484240896 00000100:00100000:1.0:1789760744.811675:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484240896 00000100:00100000:1.0:1789760744.811753:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734029342683 time 734084079909 state 0x0 ref 1 00000100:00100000:1.0:1789760744.811774:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734084340325 time 734084079909 state 0x1 ref 1 00000100:00100000:1.0:1789760744.811875:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734084340325 time 734142804006 state 0x0 ref 1 00000100:00100000:1.0:1789760744.811893:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484240896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760744.811965:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21614592, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:1.0:1789760744.811978:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.812000:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:1.0:1789760744.812005:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760744.812013:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21614592->21618687] (req 21614592->21618687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.812029:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21614592->21618687] (req 21614592->21618687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760744.853919:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21614592 -> 21618688 00010000:00010000:0.0:1789760744.853950:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21614592->21618687] (req 21614592->21618687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.853967:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21614592->21618687] (req 21614592->21618687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.853978:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21614592->21618687] (req 21614592->21618687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.853986:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760744.853992:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21614592->21618687] (req 21614592->21618687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.854002:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21614592->21618687] (req 21614592->21618687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.854027:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21614592->21618687] (req 21614592->21618687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760744.854042:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208a08 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21614592->21618687] (req 21614592->21618687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760744.854213:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484240896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42318us (43246us total) trans 4294972595 rc 0/0 00000100:00100000:0.0:1789760744.854227:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5306 00000100:00100000:0.0:1789760744.854240:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734084340325 time 734142804006 state 0x2 ref 0 00000001:00080000:2.0:1789760744.854530:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972595 is committed 00000100:00100000:0.0:1789760744.856832:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760744.856928:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484241024 00000100:00100000:2.0:1789760744.856942:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484241024 00000100:00100000:2.0:1789760744.857012:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734084340325 time 734142804006 state 0x0 ref 1 00000100:00100000:2.0:1789760744.857032:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734143064422 time 734142804006 state 0x1 ref 1 00000100:00100000:2.0:1789760744.857102:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734143064422 time 734188032377 state 0x0 ref 1 00000100:00100000:2.0:1789760744.857117:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484241024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760744.857184:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21618688, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:2.0:1789760744.857195:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.857215:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:2.0:1789760744.857221:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760744.857230:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21618688->21622783] (req 21618688->21622783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.857246:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21618688->21622783] (req 21618688->21622783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760744.905322:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972596 is committed 00002000:00010000:3.0:1789760744.905408:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21618688 -> 21622784 00010000:00010000:3.0:1789760744.905429:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21618688->21622783] (req 21618688->21622783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.905445:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21618688->21622783] (req 21618688->21622783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.905455:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21618688->21622783] (req 21618688->21622783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.905465:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.905469:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21618688->21622783] (req 21618688->21622783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.905477:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21618688->21622783] (req 21618688->21622783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.905565:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21618688->21622783] (req 21618688->21622783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.905574:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a0f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21618688->21622783] (req 21618688->21622783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760744.905695:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484241024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48573us (48867us total) trans 4294972596 rc 0/0 00000100:00100000:3.0:1789760744.905708:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5307 00000100:00100000:3.0:1789760744.905721:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734143064422 time 734188032377 state 0x2 ref 0 00000100:00100000:0.0:1789760744.910790:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760744.911170:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484241152 00000100:00100000:2.0:1789760744.911184:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484241152 00000100:00100000:2.0:1789760744.911268:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734143064422 time 734188032377 state 0x0 ref 1 00000100:00100000:2.0:1789760744.911287:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734188292793 time 734188032377 state 0x1 ref 1 00000100:00100000:2.0:1789760744.911322:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734188292793 time 734242253872 state 0x0 ref 1 00000100:00100000:2.0:1789760744.911511:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484241152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760744.911579:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21622784, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:2.0:1789760744.911590:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.911609:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:2.0:1789760744.911613:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760744.911620:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21622784->21626879] (req 21622784->21626879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.911636:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21622784->21626879] (req 21622784->21626879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760744.952182:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972597 is committed 00002000:00010000:3.0:1789760744.952433:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21622784 -> 21626880 00010000:00010000:3.0:1789760744.952455:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21622784->21626879] (req 21622784->21626879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.952471:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21622784->21626879] (req 21622784->21626879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.952498:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21622784->21626879] (req 21622784->21626879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.952506:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760744.952511:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21622784->21626879] (req 21622784->21626879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.952523:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21622784->21626879] (req 21622784->21626879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.952551:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21622784->21626879] (req 21622784->21626879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.952560:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a16 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21622784->21626879] (req 21622784->21626879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760744.952699:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484241152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41185us (41917us total) trans 4294972597 rc 0/0 00000100:00100000:3.0:1789760744.952713:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5308 00000100:00100000:3.0:1789760744.952727:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734188292793 time 734242253872 state 0x2 ref 0 00000100:00100000:0.0:1789760744.955745:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760744.955834:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484241280 00000100:00100000:3.0:1789760744.955847:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484241280 00000100:00100000:3.0:1789760744.955926:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734188292793 time 734242253872 state 0x0 ref 1 00000100:00100000:3.0:1789760744.956010:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734242514288 time 734242253872 state 0x1 ref 1 00000100:00100000:3.0:1789760744.956048:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734242514288 time 734286978924 state 0x0 ref 1 00000100:00100000:3.0:1789760744.956062:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484241280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760744.956235:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21626880, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:3.0:1789760744.956245:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.956264:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:3.0:1789760744.956270:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760744.956277:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21626880->21630975] (req 21626880->21630975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760744.956304:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21626880->21630975] (req 21626880->21630975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760744.990159:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972598 is committed 00002000:00010000:2.0:1789760744.990488:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21626880 -> 21630976 00010000:00010000:2.0:1789760744.990510:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21626880->21630975] (req 21626880->21630975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.990525:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21626880->21630975] (req 21626880->21630975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.990536:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21626880->21630975] (req 21626880->21630975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.990545:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760744.990551:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21626880->21630975] (req 21626880->21630975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.990561:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21626880->21630975] (req 21626880->21630975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.990587:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21626880->21630975] (req 21626880->21630975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760744.990596:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208a1d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21626880->21630975] (req 21626880->21630975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760744.990762:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484241280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34695us (35024us total) trans 4294972598 rc 0/0 00000100:00100000:2.0:1789760744.990779:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5309 00000100:00100000:2.0:1789760744.990791:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734242514288 time 734286978924 state 0x2 ref 0 00000100:00100000:0.0:1789760744.994979:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760744.997282:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484241408 00000100:00100000:1.0:1789760744.997313:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484241408 00000100:00100000:1.0:1789760744.997531:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734242514288 time 734286978924 state 0x0 ref 1 00000100:00100000:1.0:1789760744.997562:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734287239340 time 734286978924 state 0x1 ref 1 00000100:00100000:1.0:1789760744.997661:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734287239340 time 734328589428 state 0x0 ref 1 00000100:00100000:1.0:1789760744.997680:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484241408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760744.997754:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21630976, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:1.0:1789760744.997765:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.997787:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:1.0:1789760744.997792:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760744.997801:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21630976->21635071] (req 21630976->21635071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760744.997818:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21630976->21635071] (req 21630976->21635071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760745.032530:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972599 is committed 00002000:00010000:2.0:1789760745.032576:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21630976 -> 21635072 00010000:00010000:2.0:1789760745.032596:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21630976->21635071] (req 21630976->21635071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.032608:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21630976->21635071] (req 21630976->21635071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.032614:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21630976->21635071] (req 21630976->21635071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.032620:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760745.032624:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21630976->21635071] (req 21630976->21635071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.032632:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21630976->21635071] (req 21630976->21635071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.032654:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21630976->21635071] (req 21630976->21635071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.032661:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208a24 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21630976->21635071] (req 21630976->21635071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760745.032789:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484241408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35108us (37819us total) trans 4294972599 rc 0/0 00000100:00100000:2.0:1789760745.032800:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5310 00000100:00100000:2.0:1789760745.032812:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734287239340 time 734328589428 state 0x2 ref 0 00000100:00100000:0.0:1789760745.035058:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760745.035189:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484241536 00000100:00100000:2.0:1789760745.035202:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484241536 00000100:00100000:3.0:1789760745.035335:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734287239340 time 734328589428 state 0x0 ref 1 00000100:00100000:3.0:1789760745.035355:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734328849844 time 734328589428 state 0x1 ref 1 00000100:00100000:3.0:1789760745.035450:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734328849844 time 734366380095 state 0x0 ref 1 00000100:00100000:3.0:1789760745.035467:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484241536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760745.035531:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21635072, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:3.0:1789760745.035542:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.035559:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:3.0:1789760745.035562:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760745.035568:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21635072->21639167] (req 21635072->21639167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.035581:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21635072->21639167] (req 21635072->21639167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760745.082214:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21635072 -> 21639168 00010000:00010000:3.0:1789760745.082235:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21635072->21639167] (req 21635072->21639167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.082248:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21635072->21639167] (req 21635072->21639167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.082258:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21635072->21639167] (req 21635072->21639167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.082266:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.082270:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21635072->21639167] (req 21635072->21639167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.082279:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21635072->21639167] (req 21635072->21639167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.082309:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21635072->21639167] (req 21635072->21639167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.082317:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a2b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21635072->21639167] (req 21635072->21639167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.082415:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484241536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46947us (47365us total) trans 4294972600 rc 0/0 00000100:00100000:3.0:1789760745.082426:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5311 00000100:00100000:3.0:1789760745.082435:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734328849844 time 734366380095 state 0x2 ref 0 00000001:00080000:0.0:1789760745.084299:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972600 is committed 00000100:00100000:0.0:1789760745.091388:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760745.091577:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484241664 00000100:00100000:3.0:1789760745.091591:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484241664 00000100:00100000:3.0:1789760745.091667:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734328849844 time 734366380095 state 0x0 ref 1 00000100:00100000:3.0:1789760745.091698:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734366640511 time 734366380095 state 0x1 ref 1 00000100:00100000:3.0:1789760745.091731:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734366640511 time 734422662181 state 0x0 ref 1 00000100:00100000:3.0:1789760745.091747:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484241664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760745.091808:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21639168, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:3.0:1789760745.091817:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.091838:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:3.0:1789760745.091843:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760745.091850:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21639168->21643263] (req 21639168->21643263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.091873:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21639168->21643263] (req 21639168->21643263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760745.137249:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972601 is committed 00002000:00010000:3.0:1789760745.137366:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21639168 -> 21643264 00010000:00010000:3.0:1789760745.137387:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21639168->21643263] (req 21639168->21643263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.137400:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21639168->21643263] (req 21639168->21643263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.137410:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21639168->21643263] (req 21639168->21643263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.137418:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.137423:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21639168->21643263] (req 21639168->21643263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.137432:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21639168->21643263] (req 21639168->21643263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.137552:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21639168->21643263] (req 21639168->21643263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.137560:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a32 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21639168->21643263] (req 21639168->21643263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.137668:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484241664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45920us (46299us total) trans 4294972601 rc 0/0 00000100:00100000:3.0:1789760745.137678:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5312 00000100:00100000:3.0:1789760745.137688:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734366640511 time 734422662181 state 0x2 ref 0 00000100:00100000:2.0:1789760745.140765:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760745.140865:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484241792 00000100:00100000:3.0:1789760745.140874:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484241792 00000100:00100000:3.0:1789760745.140929:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734366640511 time 734422662181 state 0x0 ref 1 00000100:00100000:3.0:1789760745.140944:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734422922597 time 734422662181 state 0x1 ref 1 00000100:00100000:3.0:1789760745.141015:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734422922597 time 734471945246 state 0x0 ref 1 00000100:00100000:3.0:1789760745.141028:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484241792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760745.141117:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21643264, mtime: 0x6aad94e8, atime: 0x0, ctime: 0x6aad94e8, blocks: 0xa807 00010000:00010000:3.0:1789760745.141126:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.141143:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:3.0:1789760745.141147:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760745.141154:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21643264->21647359] (req 21643264->21647359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.141167:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21643264->21647359] (req 21643264->21647359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760745.175221:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972602 is committed 00002000:00010000:2.0:1789760745.175775:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21643264 -> 21647360 00002000:00010000:2.0:1789760745.175781:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760744 -> 1789760745 00002000:00010000:2.0:1789760745.175783:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760744 -> 1789760745 00010000:00010000:2.0:1789760745.175797:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21643264->21647359] (req 21643264->21647359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.175811:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21643264->21647359] (req 21643264->21647359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.175821:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21643264->21647359] (req 21643264->21647359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.175829:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760745.175834:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21643264->21647359] (req 21643264->21647359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.175843:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21643264->21647359] (req 21643264->21647359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.175864:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21643264->21647359] (req 21643264->21647359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.175872:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208a39 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21643264->21647359] (req 21643264->21647359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760745.175966:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484241792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34935us (35204us total) trans 4294972602 rc 0/0 00000100:00100000:2.0:1789760745.175976:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5313 00000100:00100000:2.0:1789760745.175985:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734422922597 time 734471945246 state 0x2 ref 0 00000100:00100000:0.0:1789760745.180960:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760745.181111:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484241920 00000100:00100000:2.0:1789760745.181121:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484241920 00000100:00100000:2.0:1789760745.181175:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734422922597 time 734471945246 state 0x0 ref 1 00000100:00100000:2.0:1789760745.181191:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734472205662 time 734471945246 state 0x1 ref 1 00000100:00100000:2.0:1789760745.181216:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734472205662 time 734512147160 state 0x0 ref 1 00000100:00100000:2.0:1789760745.181229:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484241920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760745.181279:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21647360, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:2.0:1789760745.181297:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.181314:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:2.0:1789760745.181319:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760745.181326:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21647360->21651455] (req 21647360->21651455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.181341:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21647360->21651455] (req 21647360->21651455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760745.228386:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972603 is committed 00002000:00010000:2.0:1789760745.228557:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21647360 -> 21651456 00010000:00010000:2.0:1789760745.228573:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21647360->21651455] (req 21647360->21651455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.228586:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21647360->21651455] (req 21647360->21651455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.228595:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21647360->21651455] (req 21647360->21651455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.228603:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760745.228608:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21647360->21651455] (req 21647360->21651455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.228617:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21647360->21651455] (req 21647360->21651455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.228637:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21647360->21651455] (req 21647360->21651455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.228650:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a40 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21647360->21651455] (req 21647360->21651455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760745.228751:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484241920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47519us (47795us total) trans 4294972603 rc 0/0 00000100:00100000:2.0:1789760745.228761:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5314 00000100:00100000:2.0:1789760745.228770:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734472205662 time 734512147160 state 0x2 ref 0 00000100:00100000:0.0:1789760745.239270:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760745.239521:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484242048 00000100:00100000:3.0:1789760745.239531:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484242048 00000100:00100000:3.0:1789760745.239585:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734472205662 time 734512147160 state 0x0 ref 1 00000100:00100000:3.0:1789760745.239601:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734512407576 time 734512147160 state 0x1 ref 1 00000100:00100000:3.0:1789760745.239633:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734512407576 time 734570563907 state 0x0 ref 1 00000100:00100000:3.0:1789760745.239651:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484242048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760745.239696:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21651456, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:3.0:1789760745.239705:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.239731:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:3.0:1789760745.239735:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760745.239742:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21651456->21655551] (req 21651456->21655551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.239756:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21651456->21655551] (req 21651456->21655551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760745.283650:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972604 is committed 00002000:00010000:2.0:1789760745.283747:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21651456 -> 21655552 00010000:00010000:2.0:1789760745.283762:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21651456->21655551] (req 21651456->21655551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.283775:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21651456->21655551] (req 21651456->21655551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.283784:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21651456->21655551] (req 21651456->21655551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.283791:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760745.283796:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21651456->21655551] (req 21651456->21655551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.283805:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21651456->21655551] (req 21651456->21655551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.283833:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21651456->21655551] (req 21651456->21655551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.283841:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208a47 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21651456->21655551] (req 21651456->21655551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760745.283977:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484242048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44322us (44711us total) trans 4294972604 rc 0/0 00000100:00100000:2.0:1789760745.283988:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5315 00000100:00100000:2.0:1789760745.283998:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734512407576 time 734570563907 state 0x2 ref 0 00000100:00100000:3.0:1789760745.287612:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760745.287794:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484242176 00000100:00100000:2.0:1789760745.287804:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484242176 00000100:00100000:2.0:1789760745.287858:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734512407576 time 734570563907 state 0x0 ref 1 00000100:00100000:2.0:1789760745.287875:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734570824323 time 734570563907 state 0x1 ref 1 00000100:00100000:2.0:1789760745.287944:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734570824323 time 734618874017 state 0x0 ref 1 00000100:00100000:2.0:1789760745.287958:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484242176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760745.288008:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21655552, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:2.0:1789760745.288017:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.288033:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:2.0:1789760745.288039:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760745.288045:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21655552->21659647] (req 21655552->21659647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.288058:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21655552->21659647] (req 21655552->21659647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760745.330814:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972605 is committed 00002000:00010000:0.0:1789760745.330963:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21655552 -> 21659648 00010000:00010000:0.0:1789760745.330988:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21655552->21659647] (req 21655552->21659647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.331004:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21655552->21659647] (req 21655552->21659647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.331012:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21655552->21659647] (req 21655552->21659647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.331018:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760745.331023:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21655552->21659647] (req 21655552->21659647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.331031:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21655552->21659647] (req 21655552->21659647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.331051:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21655552->21659647] (req 21655552->21659647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.331059:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a4e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21655552->21659647] (req 21655552->21659647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760745.331225:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484242176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43267us (43619us total) trans 4294972605 rc 0/0 00000100:00100000:0.0:1789760745.331236:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5316 00000100:00100000:0.0:1789760745.331245:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734570824323 time 734618874017 state 0x2 ref 0 00000100:00100000:2.0:1789760745.334562:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760745.334736:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484242304 00000100:00100000:3.0:1789760745.334746:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484242304 00000100:00100000:3.0:1789760745.334794:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734570824323 time 734618874017 state 0x0 ref 1 00000100:00100000:3.0:1789760745.334808:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734619134433 time 734618874017 state 0x1 ref 1 00000100:00100000:3.0:1789760745.334838:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734619134433 time 734665769084 state 0x0 ref 1 00000100:00100000:3.0:1789760745.334849:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484242304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760745.334896:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21659648, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:3.0:1789760745.334905:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.334920:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:3.0:1789760745.334925:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760745.334931:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21659648->21663743] (req 21659648->21663743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.334944:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21659648->21663743] (req 21659648->21663743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.356311:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:2.0:1789760745.364658:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21659648 -> 21663744 00010000:00010000:2.0:1789760745.364676:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21659648->21663743] (req 21659648->21663743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.364689:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21659648->21663743] (req 21659648->21663743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.364697:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21659648->21663743] (req 21659648->21663743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.364703:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760745.364708:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21659648->21663743] (req 21659648->21663743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.364721:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21659648->21663743] (req 21659648->21663743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.364741:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21659648->21663743] (req 21659648->21663743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.364748:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208a55 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21659648->21663743] (req 21659648->21663743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760745.364838:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484242304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29986us (30284us total) trans 4294972606 rc 0/0 00000100:00100000:2.0:1789760745.364847:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5317 00000100:00100000:2.0:1789760745.364927:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734619134433 time 734665769084 state 0x2 ref 0 00000001:00080000:2.0:1789760745.364967:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972606 is committed 00000100:00100000:0.0:1789760745.369966:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760745.370283:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484242432 00000100:00100000:1.0:1789760745.370303:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484242432 00000100:00100000:1.0:1789760745.370468:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734619134433 time 734665769084 state 0x0 ref 1 00000100:00100000:1.0:1789760745.370483:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734666029500 time 734665769084 state 0x1 ref 1 00000100:00100000:1.0:1789760745.370552:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734666029500 time 734701482751 state 0x0 ref 1 00000100:00100000:1.0:1789760745.370564:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484242432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760745.370610:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21663744, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:1.0:1789760745.370619:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760745.370634:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:1.0:1789760745.370639:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760745.370645:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21663744->21667839] (req 21663744->21667839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760745.370658:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21663744->21667839] (req 21663744->21667839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760745.411369:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972607 is committed 00002000:00010000:3.0:1789760745.411567:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21663744 -> 21667840 00010000:00010000:3.0:1789760745.411590:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21663744->21667839] (req 21663744->21667839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.411602:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21663744->21667839] (req 21663744->21667839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.411611:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21663744->21667839] (req 21663744->21667839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.411618:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.411623:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21663744->21667839] (req 21663744->21667839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.411632:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21663744->21667839] (req 21663744->21667839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.412536:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21663744->21667839] (req 21663744->21667839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.412551:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208a5c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21663744->21667839] (req 21663744->21667839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.412826:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484242432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42258us (42863us total) trans 4294972607 rc 0/0 00000100:00100000:3.0:1789760745.412838:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5318 00000100:00100000:3.0:1789760745.412848:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734666029500 time 734701482751 state 0x2 ref 0 00000100:00100000:3.0:1789760745.418243:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760745.418392:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484242560 00000100:00100000:2.0:1789760745.418403:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484242560 00000100:00100000:2.0:1789760745.418460:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734666029500 time 734701482751 state 0x0 ref 1 00000100:00100000:2.0:1789760745.418476:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734701743167 time 734701482751 state 0x1 ref 1 00000100:00100000:2.0:1789760745.418510:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734701743167 time 734749440857 state 0x0 ref 1 00000100:00100000:2.0:1789760745.418523:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484242560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760745.418580:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21667840, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:2.0:1789760745.418589:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.418606:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:2.0:1789760745.418611:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760745.418619:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21667840->21671935] (req 21667840->21671935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.418633:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21667840->21671935] (req 21667840->21671935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760745.452541:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972608 is committed 00002000:00010000:3.0:1789760745.452682:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21667840 -> 21671936 00010000:00010000:3.0:1789760745.452698:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21667840->21671935] (req 21667840->21671935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.452712:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21667840->21671935] (req 21667840->21671935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.452798:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21667840->21671935] (req 21667840->21671935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.452806:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.452811:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21667840->21671935] (req 21667840->21671935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.452821:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21667840->21671935] (req 21667840->21671935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.452841:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21667840->21671935] (req 21667840->21671935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.452849:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396208a63 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21667840->21671935] (req 21667840->21671935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.452944:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484242560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34418us (34707us total) trans 4294972608 rc 0/0 00000100:00100000:3.0:1789760745.452955:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5319 00000100:00100000:3.0:1789760745.452964:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734701743167 time 734749440857 state 0x2 ref 0 00000100:00100000:2.0:1789760745.458346:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760745.458508:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484242688 00000100:00100000:0.0:1789760745.458518:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484242688 00000100:00100000:0.0:1789760745.458579:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734701743167 time 734749440857 state 0x0 ref 1 00000100:00100000:0.0:1789760745.458595:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734749701273 time 734749440857 state 0x1 ref 1 00000100:00100000:0.0:1789760745.458621:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734749701273 time 734789552485 state 0x0 ref 1 00000100:00100000:0.0:1789760745.458634:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484242688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760745.458683:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21671936, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:0.0:1789760745.458693:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.458710:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:0.0:1789760745.458715:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760745.458722:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21671936->21676031] (req 21671936->21676031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.458737:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21671936->21676031] (req 21671936->21676031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760745.492997:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972609 is committed 00002000:00010000:3.0:1789760745.493203:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21671936 -> 21676032 00010000:00010000:3.0:1789760745.493216:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21671936->21676031] (req 21671936->21676031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.493229:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21671936->21676031] (req 21671936->21676031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.493235:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21671936->21676031] (req 21671936->21676031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.493239:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.493242:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21671936->21676031] (req 21671936->21676031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.493248:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21671936->21676031] (req 21671936->21676031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.493263:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21671936->21676031] (req 21671936->21676031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.493268:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a6a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21671936->21676031] (req 21671936->21676031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.493370:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484242688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34733us (35029us total) trans 4294972609 rc 0/0 00000100:00100000:3.0:1789760745.493481:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5320 00000100:00100000:3.0:1789760745.493495:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734749701273 time 734789552485 state 0x2 ref 0 00000100:00100000:1.0:1789760745.495583:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760745.495711:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484242816 00000100:00100000:3.0:1789760745.495721:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484242816 00000100:00100000:3.0:1789760745.495771:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734749701273 time 734789552485 state 0x0 ref 1 00000100:00100000:3.0:1789760745.495785:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734789812901 time 734789552485 state 0x1 ref 1 00000100:00100000:3.0:1789760745.495816:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734789812901 time 734826747554 state 0x0 ref 1 00000100:00100000:3.0:1789760745.495828:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484242816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760745.495874:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21676032, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:3.0:1789760745.495882:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.495896:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:3.0:1789760745.495901:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760745.495907:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21676032->21680127] (req 21676032->21680127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.495919:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21676032->21680127] (req 21676032->21680127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760745.540309:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972610 is committed 00002000:00010000:3.0:1789760745.540652:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21676032 -> 21680128 00010000:00010000:3.0:1789760745.540667:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21676032->21680127] (req 21676032->21680127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.540679:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21676032->21680127] (req 21676032->21680127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.540693:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21676032->21680127] (req 21676032->21680127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.540700:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.540704:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21676032->21680127] (req 21676032->21680127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.540712:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21676032->21680127] (req 21676032->21680127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.540730:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21676032->21680127] (req 21676032->21680127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.540737:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a71 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21676032->21680127] (req 21676032->21680127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.540824:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484242816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44992us (45243us total) trans 4294972610 rc 0/0 00000100:00100000:3.0:1789760745.540834:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5321 00000100:00100000:3.0:1789760745.540843:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734789812901 time 734826747554 state 0x2 ref 0 00000100:00100000:2.0:1789760745.544443:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760745.544662:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484242944 00000100:00100000:3.0:1789760745.544677:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484242944 00000100:00100000:3.0:1789760745.544744:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734789812901 time 734826747554 state 0x0 ref 1 00000100:00100000:3.0:1789760745.544759:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734827007970 time 734826747554 state 0x1 ref 1 00000100:00100000:3.0:1789760745.544788:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734827007970 time 734875719731 state 0x0 ref 1 00000100:00100000:3.0:1789760745.544813:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484242944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760745.544874:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21680128, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:3.0:1789760745.544883:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.544903:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:3.0:1789760745.544907:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760745.544913:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21680128->21684223] (req 21680128->21684223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.544927:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21680128->21684223] (req 21680128->21684223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760745.585451:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972611 is committed 00002000:00010000:3.0:1789760745.585668:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21680128 -> 21684224 00010000:00010000:3.0:1789760745.585682:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21680128->21684223] (req 21680128->21684223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.585694:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21680128->21684223] (req 21680128->21684223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.585702:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21680128->21684223] (req 21680128->21684223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.585709:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.585714:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21680128->21684223] (req 21680128->21684223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.585722:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21680128->21684223] (req 21680128->21684223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.585740:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21680128->21684223] (req 21680128->21684223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.585746:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208a78 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21680128->21684223] (req 21680128->21684223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.585870:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484242944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41067us (41431us total) trans 4294972611 rc 0/0 00000100:00100000:3.0:1789760745.585880:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5322 00000100:00100000:3.0:1789760745.585891:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734827007970 time 734875719731 state 0x2 ref 0 00000100:00100000:1.0:1789760745.588729:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760745.588839:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484243072 00000100:00100000:2.0:1789760745.588848:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484243072 00000100:00100000:2.0:1789760745.588895:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734827007970 time 734875719731 state 0x0 ref 1 00000100:00100000:2.0:1789760745.588910:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734875980147 time 734875719731 state 0x1 ref 1 00000100:00100000:2.0:1789760745.588938:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734875980147 time 734919868853 state 0x0 ref 1 00000100:00100000:2.0:1789760745.588950:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484243072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760745.588996:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21684224, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:2.0:1789760745.589005:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.589021:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:2.0:1789760745.589025:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760745.589031:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21684224->21688319] (req 21684224->21688319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.589050:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21684224->21688319] (req 21684224->21688319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760745.627947:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21684224 -> 21688320 00010000:00010000:3.0:1789760745.627968:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21684224->21688319] (req 21684224->21688319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.627980:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21684224->21688319] (req 21684224->21688319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.627988:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21684224->21688319] (req 21684224->21688319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.627999:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.628004:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21684224->21688319] (req 21684224->21688319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.628013:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21684224->21688319] (req 21684224->21688319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.628034:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21684224->21688319] (req 21684224->21688319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.628041:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208a7f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21684224->21688319] (req 21684224->21688319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.628185:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484243072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39231us (39459us total) trans 4294972612 rc 0/0 00000100:00100000:3.0:1789760745.628196:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5323 00000100:00100000:3.0:1789760745.628206:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734875980147 time 734919868853 state 0x2 ref 0 00000001:00080000:0.0:1789760745.632127:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972612 is committed 00000100:00100000:1.0:1789760745.632559:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760745.632743:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484243200 00000100:00100000:0.0:1789760745.632753:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484243200 00000100:00100000:0.0:1789760745.632810:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734875980147 time 734919868853 state 0x0 ref 1 00000100:00100000:0.0:1789760745.632829:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734920129269 time 734919868853 state 0x1 ref 1 00000100:00100000:0.0:1789760745.632899:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734920129269 time 734963829914 state 0x0 ref 1 00000100:00100000:0.0:1789760745.632913:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484243200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760745.632961:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21688320, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:0.0:1789760745.632970:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.632986:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=734s v=(0 0) 00010000:00010000:0.0:1789760745.632990:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760745.632997:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21688320->21692415] (req 21688320->21692415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.633011:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21688320->21692415] (req 21688320->21692415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760745.682216:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972613 is committed 00002000:00010000:0.0:1789760745.682337:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21688320 -> 21692416 00010000:00010000:0.0:1789760745.682352:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21688320->21692415] (req 21688320->21692415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.682366:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21688320->21692415] (req 21688320->21692415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.682427:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21688320->21692415] (req 21688320->21692415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.682441:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760745.682446:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21688320->21692415] (req 21688320->21692415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.682456:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21688320->21692415] (req 21688320->21692415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.682479:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21688320->21692415] (req 21688320->21692415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.682487:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208a86 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21688320->21692415] (req 21688320->21692415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760745.682691:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484243200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49776us (50132us total) trans 4294972613 rc 0/0 00000100:00100000:0.0:1789760745.682700:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5324 00000100:00100000:0.0:1789760745.682710:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734920129269 time 734963829914 state 0x2 ref 0 00000100:00100000:3.0:1789760745.689458:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760745.689683:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484243328 00000100:00100000:2.0:1789760745.689693:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484243328 00000100:00100000:2.0:1789760745.689743:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734920129269 time 734963829914 state 0x0 ref 1 00000100:00100000:2.0:1789760745.689759:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734964090330 time 734963829914 state 0x1 ref 1 00000100:00100000:2.0:1789760745.689788:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734964090330 time 735020718828 state 0x0 ref 1 00000100:00100000:2.0:1789760745.689800:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484243328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760745.689850:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21692416, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:2.0:1789760745.689857:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.689873:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:2.0:1789760745.689877:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760745.689883:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21692416->21696511] (req 21692416->21696511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.689897:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21692416->21696511] (req 21692416->21696511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760745.724984:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972614 is committed 00002000:00010000:2.0:1789760745.725202:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21692416 -> 21696512 00010000:00010000:2.0:1789760745.725219:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21692416->21696511] (req 21692416->21696511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.725232:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21692416->21696511] (req 21692416->21696511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.725241:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21692416->21696511] (req 21692416->21696511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.725248:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760745.725253:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21692416->21696511] (req 21692416->21696511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.725273:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21692416->21696511] (req 21692416->21696511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.725304:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21692416->21696511] (req 21692416->21696511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.725313:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a8d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21692416->21696511] (req 21692416->21696511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760745.725561:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484243328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35614us (35964us total) trans 4294972614 rc 0/0 00000100:00100000:2.0:1789760745.725572:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5325 00000100:00100000:2.0:1789760745.725582:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734964090330 time 735020718828 state 0x2 ref 0 00000100:00100000:1.0:1789760745.730387:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760745.730820:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484243456 00000100:00100000:2.0:1789760745.730830:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484243456 00000100:00100000:2.0:1789760745.730882:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 734964090330 time 735020718828 state 0x0 ref 1 00000100:00100000:2.0:1789760745.730896:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735020979244 time 735020718828 state 0x1 ref 1 00000100:00100000:2.0:1789760745.730927:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735020979244 time 735061858314 state 0x0 ref 1 00000100:00100000:2.0:1789760745.730939:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706c050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484243456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760745.730986:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21696512, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:2.0:1789760745.730994:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.731009:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:2.0:1789760745.731013:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760745.731019:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21696512->21700607] (req 21696512->21700607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.731032:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21696512->21700607] (req 21696512->21700607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760745.769901:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972615 is committed 00002000:00010000:2.0:1789760745.770325:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21696512 -> 21700608 00010000:00010000:2.0:1789760745.770344:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21696512->21700607] (req 21696512->21700607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.770357:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21696512->21700607] (req 21696512->21700607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.770366:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21696512->21700607] (req 21696512->21700607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.770373:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760745.770378:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21696512->21700607] (req 21696512->21700607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.770387:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21696512->21700607] (req 21696512->21700607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.770407:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21696512->21700607] (req 21696512->21700607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.770414:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208a94 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21696512->21700607] (req 21696512->21700607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760745.770808:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706c050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484243456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39864us (40430us total) trans 4294972615 rc 0/0 00000100:00100000:2.0:1789760745.770822:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5326 00000100:00100000:2.0:1789760745.770834:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735020979244 time 735061858314 state 0x2 ref 0 00000100:00100000:1.0:1789760745.783006:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760745.785580:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484243584 00000100:00100000:3.0:1789760745.785596:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484243584 00000100:00100000:3.0:1789760745.785664:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735020979244 time 735061858314 state 0x0 ref 1 00000100:00100000:3.0:1789760745.785682:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735062118730 time 735061858314 state 0x1 ref 1 00000100:00100000:3.0:1789760745.785716:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735062118730 time 735116647127 state 0x0 ref 1 00000100:00100000:3.0:1789760745.785731:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484243584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760745.785781:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21700608, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:3.0:1789760745.785790:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.785804:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:3.0:1789760745.785808:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760745.785815:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21700608->21704703] (req 21700608->21704703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.785829:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21700608->21704703] (req 21700608->21704703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760745.873589:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972616 is committed 00002000:00010000:3.0:1789760745.874279:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21700608 -> 21704704 00010000:00010000:3.0:1789760745.874306:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21700608->21704703] (req 21700608->21704703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.874320:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21700608->21704703] (req 21700608->21704703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.874329:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21700608->21704703] (req 21700608->21704703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.874341:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.874346:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21700608->21704703] (req 21700608->21704703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.874355:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21700608->21704703] (req 21700608->21704703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.874386:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21700608->21704703] (req 21700608->21704703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.874394:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208a9b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21700608->21704703] (req 21700608->21704703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.874488:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484243584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 88756us (91488us total) trans 4294972616 rc 0/0 00000100:00100000:3.0:1789760745.874499:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5327 00000100:00100000:3.0:1789760745.874509:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735062118730 time 735116647127 state 0x2 ref 0 00000100:00100000:3.0:1789760745.880225:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760745.880305:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484243712 00000100:00100000:0.0:1789760745.880315:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484243712 00000100:00100000:0.0:1789760745.880374:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735062118730 time 735116647127 state 0x0 ref 1 00000100:00100000:0.0:1789760745.880390:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735116907543 time 735116647127 state 0x1 ref 1 00000100:00100000:0.0:1789760745.880423:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735116907543 time 735211354329 state 0x0 ref 1 00000100:00100000:0.0:1789760745.880440:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484243712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760745.880484:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21704704, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:0.0:1789760745.880494:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.880509:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:0.0:1789760745.880513:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760745.880519:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21704704->21708799] (req 21704704->21708799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760745.880532:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21704704->21708799] (req 21704704->21708799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760745.917564:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972617 is committed 00002000:00010000:3.0:1789760745.917654:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21704704 -> 21708800 00010000:00010000:3.0:1789760745.917732:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21704704->21708799] (req 21704704->21708799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.917745:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21704704->21708799] (req 21704704->21708799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.917759:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21704704->21708799] (req 21704704->21708799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.917766:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.917771:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21704704->21708799] (req 21704704->21708799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.917779:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21704704->21708799] (req 21704704->21708799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.917797:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21704704->21708799] (req 21704704->21708799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.917805:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208aa2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21704704->21708799] (req 21704704->21708799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.917899:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484243712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37457us (37677us total) trans 4294972617 rc 0/0 00000100:00100000:3.0:1789760745.917908:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5328 00000100:00100000:3.0:1789760745.917917:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735116907543 time 735211354329 state 0x2 ref 0 00000100:00100000:2.0:1789760745.922533:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760745.922631:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484243840 00000100:00100000:3.0:1789760745.922642:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484243840 00000100:00100000:3.0:1789760745.922791:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735116907543 time 735211354329 state 0x0 ref 1 00000100:00100000:3.0:1789760745.922806:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735211614745 time 735211354329 state 0x1 ref 1 00000100:00100000:3.0:1789760745.922867:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735211614745 time 735253797146 state 0x0 ref 1 00000100:00100000:3.0:1789760745.922881:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484243840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760745.922928:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21708800, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:3.0:1789760745.922938:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.922953:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:3.0:1789760745.922957:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760745.922964:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21708800->21712895] (req 21708800->21712895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.922977:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21708800->21712895] (req 21708800->21712895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760745.964931:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21708800 -> 21712896 00010000:00010000:3.0:1789760745.964951:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21708800->21712895] (req 21708800->21712895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.964966:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21708800->21712895] (req 21708800->21712895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.964975:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21708800->21712895] (req 21708800->21712895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.964983:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760745.964990:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21708800->21712895] (req 21708800->21712895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.964998:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21708800->21712895] (req 21708800->21712895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.965019:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21708800->21712895] (req 21708800->21712895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760745.965027:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208aa9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21708800->21712895] (req 21708800->21712895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760745.965167:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484243840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42285us (42640us total) trans 4294972618 rc 0/0 00000100:00100000:3.0:1789760745.965177:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5329 00000100:00100000:3.0:1789760745.965187:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735211614745 time 735253797146 state 0x2 ref 0 00000001:00080000:2.0:1789760745.966103:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972618 is committed 00000100:00100000:1.0:1789760745.968739:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760745.968853:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484243968 00000100:00100000:2.0:1789760745.968862:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484243968 00000100:00100000:2.0:1789760745.968939:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735211614745 time 735253797146 state 0x0 ref 1 00000100:00100000:2.0:1789760745.968954:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735254057562 time 735253797146 state 0x1 ref 1 00000100:00100000:2.0:1789760745.968981:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735254057562 time 735299912230 state 0x0 ref 1 00000100:00100000:2.0:1789760745.968993:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484243968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760745.969043:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21712896, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:2.0:1789760745.969052:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.969103:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:2.0:1789760745.969107:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760745.969114:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21712896->21716991] (req 21712896->21716991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760745.969127:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21712896->21716991] (req 21712896->21716991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760746.007857:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21712896 -> 21716992 00010000:00010000:2.0:1789760746.007876:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21712896->21716991] (req 21712896->21716991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.007889:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21712896->21716991] (req 21712896->21716991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.007902:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21712896->21716991] (req 21712896->21716991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.007908:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760746.007913:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21712896->21716991] (req 21712896->21716991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.007921:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21712896->21716991] (req 21712896->21716991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.007940:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21712896->21716991] (req 21712896->21716991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.007948:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208ab0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21712896->21716991] (req 21712896->21716991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760746.008041:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484243968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39045us (39307us total) trans 4294972619 rc 0/0 00000100:00100000:2.0:1789760746.008052:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5330 00000100:00100000:2.0:1789760746.008061:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735254057562 time 735299912230 state 0x2 ref 0 00000001:00080000:3.0:1789760746.012324:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972619 is committed 00000100:00100000:0.0:1789760746.012524:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760746.012616:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484244096 00000100:00100000:2.0:1789760746.012626:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484244096 00000100:00100000:2.0:1789760746.012731:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735254057562 time 735299912230 state 0x0 ref 1 00000100:00100000:2.0:1789760746.012746:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735300172646 time 735299912230 state 0x1 ref 1 00000100:00100000:2.0:1789760746.012776:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735300172646 time 735343707060 state 0x0 ref 1 00000100:00100000:2.0:1789760746.012789:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484244096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760746.012836:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21716992, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:2.0:1789760746.012846:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.012862:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:2.0:1789760746.012867:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760746.012873:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21716992->21721087] (req 21716992->21721087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.012886:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21716992->21721087] (req 21716992->21721087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1789760746.015545:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760746.048500:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972620 is committed 00002000:00010000:0.0:1789760746.054532:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21716992 -> 21721088 00010000:00010000:0.0:1789760746.054551:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21716992->21721087] (req 21716992->21721087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.054563:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21716992->21721087] (req 21716992->21721087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.054571:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21716992->21721087] (req 21716992->21721087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.054578:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.054582:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21716992->21721087] (req 21716992->21721087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.054590:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21716992->21721087] (req 21716992->21721087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.054609:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21716992->21721087] (req 21716992->21721087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.054616:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208ab7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21716992->21721087] (req 21716992->21721087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.054717:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484244096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41926us (42194us total) trans 4294972620 rc 0/0 00000100:00100000:0.0:1789760746.054728:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5331 00000100:00100000:0.0:1789760746.054737:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735300172646 time 735343707060 state 0x2 ref 0 00000100:00100000:0.0:1789760746.058951:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760746.059441:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484244224 00000100:00100000:3.0:1789760746.059453:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484244224 00000100:00100000:3.0:1789760746.059510:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735300172646 time 735343707060 state 0x0 ref 1 00000100:00100000:3.0:1789760746.059525:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735343967476 time 735343707060 state 0x1 ref 1 00000100:00100000:3.0:1789760746.059602:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735343967476 time 735390531833 state 0x0 ref 1 00000100:00100000:3.0:1789760746.059615:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484244224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760746.059665:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21721088, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:3.0:1789760746.059673:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.059688:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:3.0:1789760746.059692:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760746.059699:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21721088->21725183] (req 21721088->21725183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.059713:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21721088->21725183] (req 21721088->21725183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760746.098362:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972621 is committed 00002000:00010000:3.0:1789760746.098470:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21721088 -> 21725184 00010000:00010000:3.0:1789760746.098485:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21721088->21725183] (req 21721088->21725183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.098497:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21721088->21725183] (req 21721088->21725183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.098506:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21721088->21725183] (req 21721088->21725183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.098513:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760746.098517:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21721088->21725183] (req 21721088->21725183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.098526:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21721088->21725183] (req 21721088->21725183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.098546:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21721088->21725183] (req 21721088->21725183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.098553:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208abe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21721088->21725183] (req 21721088->21725183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760746.098642:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484244224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39025us (39695us total) trans 4294972621 rc 0/0 00000100:00100000:3.0:1789760746.098653:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5332 00000100:00100000:3.0:1789760746.098666:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735343967476 time 735390531833 state 0x2 ref 0 00000100:00100000:1.0:1789760746.103536:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760746.106362:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484244352 00000100:00100000:3.0:1789760746.106376:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484244352 00000100:00100000:3.0:1789760746.106434:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735343967476 time 735390531833 state 0x0 ref 1 00000100:00100000:3.0:1789760746.106449:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735390792249 time 735390531833 state 0x1 ref 1 00000100:00100000:3.0:1789760746.106472:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735390792249 time 735437402894 state 0x0 ref 1 00000100:00100000:3.0:1789760746.106485:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484244352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760746.106532:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21725184, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:3.0:1789760746.106541:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.106557:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:3.0:1789760746.106562:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760746.106568:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21725184->21729279] (req 21725184->21729279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.106581:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21725184->21729279] (req 21725184->21729279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760746.134153:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972622 is committed 00002000:00010000:2.0:1789760746.134520:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21725184 -> 21729280 00010000:00010000:2.0:1789760746.134535:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21725184->21729279] (req 21725184->21729279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.134547:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21725184->21729279] (req 21725184->21729279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.134557:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21725184->21729279] (req 21725184->21729279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.134570:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760746.134575:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21725184->21729279] (req 21725184->21729279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.134584:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21725184->21729279] (req 21725184->21729279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.134603:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21725184->21729279] (req 21725184->21729279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.134610:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ac5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21725184->21729279] (req 21725184->21729279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760746.134698:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484244352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28211us (31166us total) trans 4294972622 rc 0/0 00000100:00100000:2.0:1789760746.134707:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5333 00000100:00100000:2.0:1789760746.134717:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735390792249 time 735437402894 state 0x2 ref 0 00000100:00100000:0.0:1789760746.140219:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760746.140302:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484244480 00000100:00100000:2.0:1789760746.140312:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484244480 00000100:00100000:2.0:1789760746.140367:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735390792249 time 735437402894 state 0x0 ref 1 00000100:00100000:2.0:1789760746.140384:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735437663310 time 735437402894 state 0x1 ref 1 00000100:00100000:2.0:1789760746.140458:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735437663310 time 735471387749 state 0x0 ref 1 00000100:00100000:2.0:1789760746.140474:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484244480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760746.140605:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21729280, mtime: 0x6aad94e9, atime: 0x0, ctime: 0x6aad94e9, blocks: 0xa807 00010000:00010000:2.0:1789760746.140614:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.140629:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:2.0:1789760746.140633:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760746.140639:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21729280->21733375] (req 21729280->21733375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.140653:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21729280->21733375] (req 21729280->21733375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760746.201616:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972623 is committed 00002000:00010000:0.0:1789760746.203079:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21729280 -> 21733376 00002000:00010000:0.0:1789760746.203087:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760745 -> 1789760746 00002000:00010000:0.0:1789760746.203090:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760745 -> 1789760746 00010000:00010000:0.0:1789760746.203104:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21729280->21733375] (req 21729280->21733375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.203118:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21729280->21733375] (req 21729280->21733375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.203133:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21729280->21733375] (req 21729280->21733375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.203141:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.203146:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21729280->21733375] (req 21729280->21733375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.203156:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21729280->21733375] (req 21729280->21733375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.203177:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21729280->21733375] (req 21729280->21733375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.203185:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208acc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21729280->21733375] (req 21729280->21733375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.203275:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484244480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62798us (63060us total) trans 4294972623 rc 0/0 00000100:00100000:0.0:1789760746.203297:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5334 00000100:00100000:0.0:1789760746.203307:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735437663310 time 735471387749 state 0x2 ref 0 00000100:00100000:0.0:1789760746.207988:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760746.208527:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484244608 00000100:00100000:2.0:1789760746.208538:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484244608 00000100:00100000:2.0:1789760746.208592:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735437663310 time 735471387749 state 0x0 ref 1 00000100:00100000:2.0:1789760746.208608:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735471648165 time 735471387749 state 0x1 ref 1 00000100:00100000:2.0:1789760746.208643:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735471648165 time 735539573585 state 0x0 ref 1 00000100:00100000:2.0:1789760746.208666:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484244608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760746.208725:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21733376, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:2.0:1789760746.208734:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.208750:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:2.0:1789760746.208754:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760746.208760:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21733376->21737471] (req 21733376->21737471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.208775:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21733376->21737471] (req 21733376->21737471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760746.267418:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972624 is committed 00002000:00010000:0.0:1789760746.267528:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21733376 -> 21737472 00010000:00010000:0.0:1789760746.267543:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21733376->21737471] (req 21733376->21737471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.267563:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21733376->21737471] (req 21733376->21737471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.267572:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21733376->21737471] (req 21733376->21737471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.267579:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.267584:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21733376->21737471] (req 21733376->21737471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.267594:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21733376->21737471] (req 21733376->21737471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.267614:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21733376->21737471] (req 21733376->21737471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.267622:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ad3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21733376->21737471] (req 21733376->21737471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.267745:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484244608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59087us (59764us total) trans 4294972624 rc 0/0 00000100:00100000:0.0:1789760746.267755:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5335 00000100:00100000:0.0:1789760746.267765:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735471648165 time 735539573585 state 0x2 ref 0 00000100:00100000:1.0:1789760746.270634:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760746.270719:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484244736 00000100:00100000:3.0:1789760746.270728:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484244736 00000100:00100000:3.0:1789760746.270784:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735471648165 time 735539573585 state 0x0 ref 1 00000100:00100000:3.0:1789760746.270800:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735539834001 time 735539573585 state 0x1 ref 1 00000100:00100000:3.0:1789760746.270872:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735539834001 time 735601801404 state 0x0 ref 1 00000100:00100000:3.0:1789760746.270886:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484244736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760746.270933:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21737472, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:3.0:1789760746.270942:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.270959:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:3.0:1789760746.270964:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760746.270970:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21737472->21741567] (req 21737472->21741567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.270985:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21737472->21741567] (req 21737472->21741567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760746.308388:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972625 is committed 00002000:00010000:2.0:1789760746.308494:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21737472 -> 21741568 00010000:00010000:2.0:1789760746.308509:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21737472->21741567] (req 21737472->21741567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.308523:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21737472->21741567] (req 21737472->21741567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.308533:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21737472->21741567] (req 21737472->21741567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.308541:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760746.308546:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21737472->21741567] (req 21737472->21741567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.308556:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21737472->21741567] (req 21737472->21741567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.308576:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21737472->21741567] (req 21737472->21741567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.308584:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ada lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21737472->21741567] (req 21737472->21741567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760746.308681:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484244736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37793us (38050us total) trans 4294972625 rc 0/0 00000100:00100000:2.0:1789760746.308692:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5336 00000100:00100000:2.0:1789760746.308703:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735539834001 time 735601801404 state 0x2 ref 0 00000100:00100000:1.0:1789760746.316230:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760746.316306:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484244864 00000100:00100000:3.0:1789760746.316316:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484244864 00000100:00100000:3.0:1789760746.316383:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735539834001 time 735601801404 state 0x0 ref 1 00000100:00100000:3.0:1789760746.316398:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735602061820 time 735601801404 state 0x1 ref 1 00000100:00100000:3.0:1789760746.316428:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735602061820 time 735647358801 state 0x0 ref 1 00000100:00100000:3.0:1789760746.316440:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484244864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760746.316489:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21741568, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:3.0:1789760746.316498:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.316513:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:3.0:1789760746.316518:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760746.316529:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21741568->21745663] (req 21741568->21745663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.316542:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21741568->21745663] (req 21741568->21745663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.380694:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:0.0:1789760746.396711:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21741568 -> 21745664 00010000:00010000:0.0:1789760746.396743:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21741568->21745663] (req 21741568->21745663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.396756:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21741568->21745663] (req 21741568->21745663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.396765:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21741568->21745663] (req 21741568->21745663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.396772:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.396777:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21741568->21745663] (req 21741568->21745663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.396786:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21741568->21745663] (req 21741568->21745663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.396807:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21741568->21745663] (req 21741568->21745663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.396815:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ae1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21741568->21745663] (req 21741568->21745663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.396910:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484244864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 80468us (80684us total) trans 4294972626 rc 0/0 00000100:00100000:0.0:1789760746.396921:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5337 00000100:00100000:0.0:1789760746.396930:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735602061820 time 735647358801 state 0x2 ref 0 00000001:00080000:0.0:1789760746.397031:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972626 is committed 00000100:00100000:0.0:1789760746.403532:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760746.403634:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484244992 00000100:00100000:2.0:1789760746.403646:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484244992 00000100:00100000:2.0:1789760746.403702:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735602061820 time 735647358801 state 0x0 ref 1 00000100:00100000:2.0:1789760746.403718:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735647619217 time 735647358801 state 0x1 ref 1 00000100:00100000:2.0:1789760746.403758:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735647619217 time 735734688747 state 0x0 ref 1 00000100:00100000:2.0:1789760746.403771:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484244992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760746.403820:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21745664, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:2.0:1789760746.403829:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.403846:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:2.0:1789760746.403850:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760746.403856:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21745664->21749759] (req 21745664->21749759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.403875:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21745664->21749759] (req 21745664->21749759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760746.460449:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972627 is committed 00002000:00010000:0.0:1789760746.460628:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21745664 -> 21749760 00010000:00010000:0.0:1789760746.460644:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21745664->21749759] (req 21745664->21749759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.460657:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21745664->21749759] (req 21745664->21749759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.460665:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21745664->21749759] (req 21745664->21749759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.460672:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.460676:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21745664->21749759] (req 21745664->21749759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.460686:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21745664->21749759] (req 21745664->21749759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.460705:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21745664->21749759] (req 21745664->21749759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.460713:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ae8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21745664->21749759] (req 21745664->21749759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.460839:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484244992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57065us (57313us total) trans 4294972627 rc 0/0 00000100:00100000:0.0:1789760746.460849:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5338 00000100:00100000:0.0:1789760746.460859:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735647619217 time 735734688747 state 0x2 ref 0 00000100:00100000:2.0:1789760746.465155:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760746.465252:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484245120 00000100:00100000:0.0:1789760746.465263:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484245120 00000100:00100000:0.0:1789760746.465337:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735647619217 time 735734688747 state 0x0 ref 1 00000100:00100000:0.0:1789760746.465354:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735734949163 time 735734688747 state 0x1 ref 1 00000100:00100000:0.0:1789760746.465385:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735734949163 time 735796316293 state 0x0 ref 1 00000100:00100000:0.0:1789760746.465399:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484245120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760746.465453:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21749760, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:0.0:1789760746.465461:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.465477:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:0.0:1789760746.465481:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760746.465487:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21749760->21753855] (req 21749760->21753855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.465501:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21749760->21753855] (req 21749760->21753855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760746.498484:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972628 is committed 00002000:00010000:2.0:1789760746.498636:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21749760 -> 21753856 00010000:00010000:2.0:1789760746.498654:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21749760->21753855] (req 21749760->21753855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.498666:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21749760->21753855] (req 21749760->21753855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.498675:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21749760->21753855] (req 21749760->21753855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.498682:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760746.498687:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21749760->21753855] (req 21749760->21753855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.498696:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21749760->21753855] (req 21749760->21753855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.498715:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21749760->21753855] (req 21749760->21753855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.498722:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208aef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21749760->21753855] (req 21749760->21753855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760746.498817:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484245120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33416us (33667us total) trans 4294972628 rc 0/0 00000100:00100000:2.0:1789760746.498827:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5339 00000100:00100000:2.0:1789760746.498837:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735734949163 time 735796316293 state 0x2 ref 0 00000100:00100000:0.0:1789760746.504692:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760746.504841:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484245248 00000100:00100000:3.0:1789760746.504851:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484245248 00000100:00100000:3.0:1789760746.504902:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735734949163 time 735796316293 state 0x0 ref 1 00000100:00100000:3.0:1789760746.504918:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735796576709 time 735796316293 state 0x1 ref 1 00000100:00100000:3.0:1789760746.504948:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735796576709 time 735835879211 state 0x0 ref 1 00000100:00100000:3.0:1789760746.504961:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484245248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760746.505008:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21753856, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:3.0:1789760746.505017:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.505033:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:3.0:1789760746.505037:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760746.505043:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21753856->21757951] (req 21753856->21757951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.505056:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21753856->21757951] (req 21753856->21757951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760746.544490:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972629 is committed 00002000:00010000:0.0:1789760746.549618:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21753856 -> 21757952 00010000:00010000:0.0:1789760746.549640:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21753856->21757951] (req 21753856->21757951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.549654:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21753856->21757951] (req 21753856->21757951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.549663:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21753856->21757951] (req 21753856->21757951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.549671:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.549676:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21753856->21757951] (req 21753856->21757951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.549685:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21753856->21757951] (req 21753856->21757951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.549703:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21753856->21757951] (req 21753856->21757951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.549711:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208af6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21753856->21757951] (req 21753856->21757951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.549871:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484245248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44907us (45183us total) trans 4294972629 rc 0/0 00000100:00100000:0.0:1789760746.549881:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5340 00000100:00100000:0.0:1789760746.549892:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735796576709 time 735835879211 state 0x2 ref 0 00000100:00100000:0.0:1789760746.561911:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760746.562118:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484245376 00000100:00100000:3.0:1789760746.562130:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484245376 00000100:00100000:2.0:1789760746.562236:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735796576709 time 735835879211 state 0x0 ref 1 00000100:00100000:2.0:1789760746.562251:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735836139627 time 735835879211 state 0x1 ref 1 00000100:00100000:2.0:1789760746.562293:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735836139627 time 735893223713 state 0x0 ref 1 00000100:00100000:2.0:1789760746.562305:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484245376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760746.562354:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21757952, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:2.0:1789760746.562363:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.562378:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:2.0:1789760746.562382:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760746.562387:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21757952->21762047] (req 21757952->21762047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.562400:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21757952->21762047] (req 21757952->21762047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760746.587721:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21757952 -> 21762048 00010000:00010000:2.0:1789760746.587740:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21757952->21762047] (req 21757952->21762047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.587753:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21757952->21762047] (req 21757952->21762047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.587761:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21757952->21762047] (req 21757952->21762047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.587769:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760746.587773:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21757952->21762047] (req 21757952->21762047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.587783:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21757952->21762047] (req 21757952->21762047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.587802:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21757952->21762047] (req 21757952->21762047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.587809:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208afd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21757952->21762047] (req 21757952->21762047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760746.587896:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484245376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25589us (25993us total) trans 4294972630 rc 0/0 00000100:00100000:2.0:1789760746.587913:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5341 00000100:00100000:2.0:1789760746.587923:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735836139627 time 735893223713 state 0x2 ref 0 00000001:00080000:3.0:1789760746.587993:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972630 is committed 00000100:00100000:2.0:1789760746.591503:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760746.591630:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484245504 00000100:00100000:0.0:1789760746.591640:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484245504 00000100:00100000:0.0:1789760746.591700:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735836139627 time 735893223713 state 0x0 ref 1 00000100:00100000:0.0:1789760746.591716:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735893484129 time 735893223713 state 0x1 ref 1 00000100:00100000:0.0:1789760746.591775:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735893484129 time 735922705427 state 0x0 ref 1 00000100:00100000:0.0:1789760746.591789:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484245504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760746.591847:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21762048, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:0.0:1789760746.591856:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.591871:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:0.0:1789760746.591875:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760746.591881:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21762048->21766143] (req 21762048->21766143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.591894:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21762048->21766143] (req 21762048->21766143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760746.625231:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972631 is committed 00002000:00010000:0.0:1789760746.629680:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21762048 -> 21766144 00010000:00010000:0.0:1789760746.629701:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21762048->21766143] (req 21762048->21766143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.629719:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21762048->21766143] (req 21762048->21766143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.629728:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21762048->21766143] (req 21762048->21766143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.629734:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.629739:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21762048->21766143] (req 21762048->21766143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.629748:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21762048->21766143] (req 21762048->21766143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.629768:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21762048->21766143] (req 21762048->21766143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.629776:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b04 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21762048->21766143] (req 21762048->21766143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.629881:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484245504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38090us (38380us total) trans 4294972631 rc 0/0 00000100:00100000:0.0:1789760746.629891:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5342 00000100:00100000:0.0:1789760746.629900:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735893484129 time 735922705427 state 0x2 ref 0 00000100:00100000:0.0:1789760746.633548:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760746.634107:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484245632 00000100:00100000:2.0:1789760746.634117:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484245632 00000100:00100000:2.0:1789760746.634172:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735893484129 time 735922705427 state 0x0 ref 1 00000100:00100000:2.0:1789760746.634188:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735922965843 time 735922705427 state 0x1 ref 1 00000100:00100000:2.0:1789760746.634214:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735922965843 time 735965144653 state 0x0 ref 1 00000100:00100000:2.0:1789760746.634226:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484245632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760746.634288:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21766144, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:2.0:1789760746.634298:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.634313:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=735s v=(0 0) 00010000:00010000:2.0:1789760746.634318:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760746.634323:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21766144->21770239] (req 21766144->21770239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.634337:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21766144->21770239] (req 21766144->21770239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760746.699710:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21766144 -> 21770240 00010000:00010000:0.0:1789760746.699731:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21766144->21770239] (req 21766144->21770239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.699744:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21766144->21770239] (req 21766144->21770239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.699754:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21766144->21770239] (req 21766144->21770239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.699762:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.699767:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21766144->21770239] (req 21766144->21770239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.699776:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21766144->21770239] (req 21766144->21770239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.699798:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21766144->21770239] (req 21766144->21770239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.699806:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208b0b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21766144->21770239] (req 21766144->21770239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.699892:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484245632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65663us (66349us total) trans 4294972632 rc 0/0 00000100:00100000:0.0:1789760746.699902:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5343 00000100:00100000:0.0:1789760746.699912:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735922965843 time 735965144653 state 0x2 ref 0 00000001:00080000:3.0:1789760746.699989:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972632 is committed 00000100:00100000:2.0:1789760746.704553:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760746.704635:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484245760 00000100:00100000:0.0:1789760746.704645:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484245760 00000100:00100000:0.0:1789760746.704702:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735922965843 time 735965144653 state 0x0 ref 1 00000100:00100000:0.0:1789760746.704718:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735965405069 time 735965144653 state 0x1 ref 1 00000100:00100000:0.0:1789760746.704750:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735965405069 time 736035680627 state 0x0 ref 1 00000100:00100000:0.0:1789760746.704764:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484245760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760746.704814:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21770240, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:0.0:1789760746.704823:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.704838:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:0.0:1789760746.704844:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760746.704851:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21770240->21774335] (req 21770240->21774335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.704865:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21770240->21774335] (req 21770240->21774335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760746.746042:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21770240 -> 21774336 00010000:00010000:1.0:1789760746.746061:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21770240->21774335] (req 21770240->21774335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.746104:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21770240->21774335] (req 21770240->21774335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.746113:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21770240->21774335] (req 21770240->21774335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.746121:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760746.746125:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21770240->21774335] (req 21770240->21774335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.746135:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21770240->21774335] (req 21770240->21774335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.746154:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21770240->21774335] (req 21770240->21774335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.746162:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b12 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21770240->21774335] (req 21770240->21774335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760746.746253:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484245760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41488us (41705us total) trans 4294972633 rc 0/0 00000100:00100000:1.0:1789760746.746263:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5344 00000100:00100000:1.0:1789760746.746273:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735965405069 time 736035680627 state 0x2 ref 0 00000001:00080000:3.0:1789760746.746545:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972633 is committed 00000100:00100000:2.0:1789760746.753920:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760746.754095:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484245888 00000100:00100000:0.0:1789760746.754105:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484245888 00000100:00100000:0.0:1789760746.754163:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 735965405069 time 736035680627 state 0x0 ref 1 00000100:00100000:0.0:1789760746.754178:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736035941043 time 736035680627 state 0x1 ref 1 00000100:00100000:0.0:1789760746.754215:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736035941043 time 736085146161 state 0x0 ref 1 00000100:00100000:0.0:1789760746.754229:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484245888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760746.754293:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21774336, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:0.0:1789760746.754303:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.754319:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:0.0:1789760746.754324:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760746.754330:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21774336->21778431] (req 21774336->21778431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.754344:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21774336->21778431] (req 21774336->21778431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760746.801650:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972634 is committed 00002000:00010000:3.0:1789760746.801865:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21774336 -> 21778432 00010000:00010000:3.0:1789760746.801880:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21774336->21778431] (req 21774336->21778431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.801893:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21774336->21778431] (req 21774336->21778431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.801907:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21774336->21778431] (req 21774336->21778431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.801914:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760746.801919:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21774336->21778431] (req 21774336->21778431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.801928:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21774336->21778431] (req 21774336->21778431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.801957:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21774336->21778431] (req 21774336->21778431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.801965:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208b19 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21774336->21778431] (req 21774336->21778431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760746.802097:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484245888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47834us (48150us total) trans 4294972634 rc 0/0 00000100:00100000:3.0:1789760746.802108:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5345 00000100:00100000:3.0:1789760746.802117:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736035941043 time 736085146161 state 0x2 ref 0 00000100:00100000:0.0:1789760746.809917:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760746.810022:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484246016 00000100:00100000:2.0:1789760746.810031:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484246016 00000100:00100000:2.0:1789760746.810119:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736035941043 time 736085146161 state 0x0 ref 1 00000100:00100000:2.0:1789760746.810135:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736085406577 time 736085146161 state 0x1 ref 1 00000100:00100000:2.0:1789760746.810173:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736085406577 time 736141097096 state 0x0 ref 1 00000100:00100000:2.0:1789760746.810187:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484246016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760746.810235:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21778432, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:2.0:1789760746.810244:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.810260:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:2.0:1789760746.810264:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760746.810270:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21778432->21782527] (req 21778432->21782527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760746.810294:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21778432->21782527] (req 21778432->21782527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760746.848577:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21778432 -> 21782528 00010000:00010000:0.0:1789760746.848597:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21778432->21782527] (req 21778432->21782527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.848611:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21778432->21782527] (req 21778432->21782527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.848619:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21778432->21782527] (req 21778432->21782527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.848632:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.848637:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21778432->21782527] (req 21778432->21782527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.848646:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21778432->21782527] (req 21778432->21782527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.848678:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21778432->21782527] (req 21778432->21782527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.848687:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208b20 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21778432->21782527] (req 21778432->21782527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.848778:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484246016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38590us (38866us total) trans 4294972635 rc 0/0 00000100:00100000:0.0:1789760746.848794:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5346 00000100:00100000:0.0:1789760746.848803:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736085406577 time 736141097096 state 0x2 ref 0 00000001:00080000:3.0:1789760746.848875:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972635 is committed 00000100:00100000:2.0:1789760746.859833:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760746.859927:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484246144 00000100:00100000:0.0:1789760746.859937:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484246144 00000100:00100000:0.0:1789760746.859995:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736085406577 time 736141097096 state 0x0 ref 1 00000100:00100000:0.0:1789760746.860010:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736141357512 time 736141097096 state 0x1 ref 1 00000100:00100000:0.0:1789760746.860041:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736141357512 time 736190972264 state 0x0 ref 1 00000100:00100000:0.0:1789760746.860059:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484246144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760746.860144:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21782528, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:0.0:1789760746.860153:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.860168:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:0.0:1789760746.860172:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760746.860178:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21782528->21786623] (req 21782528->21786623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.860191:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21782528->21786623] (req 21782528->21786623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760746.892772:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508922496:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.892841:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760746.892887:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508923008:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.892935:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760746.892965:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508923520:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.892987:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760746.893003:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508924032:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893020:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760746.893041:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [2->3] 00000100:00100000:2.0:1789760746.893046:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508922880:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893178:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760746.893196:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508923392:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893218:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760746.893234:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508923904:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893253:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760746.893293:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508922496 00000100:00100000:2.0:1789760746.893301:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508922496 00000100:00100000:2.0:1789760746.893533:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 480 00000100:00100000:2.0:1789760746.893543:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b4c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508922496:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893592:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b4c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508922496:12345-0@lo:400:kworker.0 Request processed in 50us (750us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760746.893598:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 480 00000100:00100000:2.0:1789760746.893635:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508923008 00000100:00100000:2.0:1789760746.893639:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508923008 00000100:00100000:2.0:1789760746.893664:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 443 00000100:00100000:2.0:1789760746.893669:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508923008:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893703:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508923008:12345-0@lo:400:kworker.0 Request processed in 33us (767us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760746.893708:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 443 00000100:00100000:2.0:1789760746.893724:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508923520 00000100:00100000:2.0:1789760746.893727:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508923520 00000100:00100000:2.0:1789760746.893757:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508924032 00000100:00100000:2.0:1789760746.893760:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508924032 00000100:00100000:2.0:1789760746.893773:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508923392 00000100:00100000:2.0:1789760746.893776:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508923392 00000100:00100000:2.0:1789760746.893788:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508923904 00000100:00100000:2.0:1789760746.893790:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508923904 00000100:00100000:2.0:1789760746.893801:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 323 00000100:00100000:2.0:1789760746.893807:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508923520:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893837:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508923520:12345-0@lo:400:kworker.0 Request processed in 30us (848us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760746.893842:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 323 00000100:00100000:2.0:1789760746.893853:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 324 00000100:00100000:2.0:1789760746.893859:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508924032:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893883:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508924032:12345-0@lo:400:kworker.0 Request processed in 26us (862us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760746.893889:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 324 00000100:00100000:2.0:1789760746.893899:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 325 00000100:00100000:2.0:1789760746.893904:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508923392:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893929:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508923392:12345-0@lo:400:kworker.0 Request processed in 26us (711us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760746.893935:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 325 00000100:00100000:2.0:1789760746.893958:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 326 00000100:00100000:2.0:1789760746.893965:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508923904:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.893991:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9471880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508923904:12345-0@lo:400:kworker.0 Request processed in 26us (737us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760746.893996:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 326 00000100:00100000:2.0:1789760746.894499:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508923392:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.894517:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508923904:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.894527:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508922496:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.894536:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98da680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508923008:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.894545:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508923520:0@lo:400:kworker.0 00000100:00100000:2.0:1789760746.894554:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508924032:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.897663:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508922624:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.897734:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760746.897781:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508923136:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.897805:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760746.897827:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508923648:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.897896:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760746.897911:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9470e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508924160:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760746.897932:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760746.897965:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1789760746.897972:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508922752:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.897995:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760746.898011:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508923264:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898029:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760746.898045:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508923776:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898086:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760746.898120:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508922880 00000100:00100000:1.0:1789760746.898127:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508922880 00000100:00100000:1.0:1789760746.898159:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508922624 00000100:00100000:1.0:1789760746.898162:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508922624 00000100:00100000:1.0:1789760746.898172:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508923136 00000100:00100000:1.0:1789760746.898174:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508923136 00000100:00100000:1.0:1789760746.898187:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508923648 00000100:00100000:1.0:1789760746.898189:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508923648 00000100:00100000:1.0:1789760746.898200:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508923264 00000100:00100000:1.0:1789760746.898201:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508923264 00000100:00100000:1.0:1789760746.898212:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508923776 00000100:00100000:1.0:1789760746.898214:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508923776 00000100:00100000:1.0:1789760746.898228:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 925 00000100:00100000:1.0:1789760746.898235:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508922880:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898294:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508922880:12345-0@lo:400:kworker.0 Request processed in 59us (5115us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760746.898302:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 925 00000100:00100000:1.0:1789760746.898327:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 926 00000100:00100000:1.0:1789760746.898333:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508922624:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898362:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508922624:12345-0@lo:400:kworker.0 Request processed in 30us (628us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760746.898368:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 926 00000100:00100000:1.0:1789760746.898379:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 927 00000100:00100000:1.0:1789760746.898384:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508923136:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898408:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508923136:12345-0@lo:400:kworker.0 Request processed in 24us (602us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760746.898413:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 927 00000100:00100000:1.0:1789760746.898423:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 928 00000100:00100000:1.0:1789760746.898428:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508923648:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898452:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508923648:12345-0@lo:400:kworker.0 Request processed in 24us (556us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760746.898457:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 928 00000100:00100000:1.0:1789760746.898467:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 929 00000100:00100000:1.0:1789760746.898473:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508923264:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898497:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508923264:12345-0@lo:400:kworker.0 Request processed in 24us (467us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760746.898502:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 929 00000100:00100000:1.0:1789760746.898511:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 930 00000100:00100000:1.0:1789760746.898517:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c44700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508923776:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898540:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c44700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508923776:12345-0@lo:400:kworker.0 Request processed in 24us (454us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760746.898546:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 930 00000100:00100000:1.0:1789760746.898572:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508922752 00000100:00100000:1.0:1789760746.898575:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508922752 00000100:00100000:1.0:1789760746.898598:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 444 00000100:00100000:1.0:1789760746.898604:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508922752:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898633:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508922752:12345-0@lo:400:kworker.0 Request processed in 30us (637us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760746.898638:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 444 00000100:00080000:1.0:1789760746.898653:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:1.0:1789760746.898657:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:1.0:1789760746.898659:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:1.0:1789760746.898662:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1789760746.898679:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508924160 00000100:00100000:1.0:1789760746.898682:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508924160 00000100:00100000:1.0:1789760746.898704:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 384 00000100:00100000:1.0:1789760746.898710:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508924160:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:1.0:1789760746.898737:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760746.898766:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508924160:12345-0@lo:13:osp-pre-1-0.0 Request processed in 56us (833us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760746.898772:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 384 00000100:00100000:1.0:1789760746.898813:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508922752:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898822:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98daa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508923264:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898830:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508923776:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898838:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508922624:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898847:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98db480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508923136:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898854:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508923648:0@lo:400:kworker.0 00000100:00100000:1.0:1789760746.898867:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9470e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508924160:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760746.898911:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98dbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508922880:0@lo:400:kworker.0 00000100:00100000:0.0:1789760746.906576:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5082a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508924288:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760746.906644:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760746.906728:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508924288 00000100:00100000:1.0:1789760746.906735:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508924288 00000100:00100000:1.0:1789760746.906781:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 385 00000100:00100000:1.0:1789760746.906790:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508924288:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:1.0:1789760746.906814:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760746.906860:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508924288:12345-0@lo:13:osp-pre-1-1.0 Request processed in 71us (215us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760746.906866:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 385 00000100:00100000:1.0:1789760746.906912:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5082a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508924288:0@lo:13:osp-pre-1-1.0 00000001:00080000:1.0:1789760746.915611:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972636 is committed 00002000:00010000:0.0:1789760746.915865:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21782528 -> 21786624 00010000:00010000:0.0:1789760746.915880:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21782528->21786623] (req 21782528->21786623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.915893:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21782528->21786623] (req 21782528->21786623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.915901:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21782528->21786623] (req 21782528->21786623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.915914:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760746.915918:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21782528->21786623] (req 21782528->21786623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.915927:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21782528->21786623] (req 21782528->21786623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.915954:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21782528->21786623] (req 21782528->21786623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.915962:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b27 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21782528->21786623] (req 21782528->21786623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760746.916097:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484246144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55994us (56226us total) trans 4294972636 rc 0/0 00000100:00100000:0.0:1789760746.916109:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5347 00000100:00100000:0.0:1789760746.916118:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736141357512 time 736190972264 state 0x2 ref 0 00000100:00100000:1.0:1789760746.921035:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760746.921282:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484246272 00000100:00100000:0.0:1789760746.921294:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484246272 00000100:00100000:0.0:1789760746.921415:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736141357512 time 736190972264 state 0x0 ref 1 00000100:00100000:0.0:1789760746.921430:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736191232680 time 736190972264 state 0x1 ref 1 00000100:00100000:0.0:1789760746.921492:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736191232680 time 736252422913 state 0x0 ref 1 00000100:00100000:0.0:1789760746.921509:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb5880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484246272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760746.921556:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21786624, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:0.0:1789760746.921564:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.921580:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:0.0:1789760746.921584:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760746.921590:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21786624->21790719] (req 21786624->21790719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760746.921604:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21786624->21790719] (req 21786624->21790719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760746.943946:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972637 is committed 00002000:00010000:1.0:1789760746.943984:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21786624 -> 21790720 00010000:00010000:1.0:1789760746.943999:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21786624->21790719] (req 21786624->21790719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.944019:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21786624->21790719] (req 21786624->21790719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.944028:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21786624->21790719] (req 21786624->21790719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.944034:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760746.944039:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21786624->21790719] (req 21786624->21790719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.944047:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21786624->21790719] (req 21786624->21790719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.944102:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21786624->21790719] (req 21786624->21790719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760746.944111:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396208b2e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21786624->21790719] (req 21786624->21790719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760746.944382:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb5880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484246272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 22872us (23356us total) trans 4294972637 rc 0/0 00000100:00100000:2.0:1789760746.944393:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5348 00000100:00100000:2.0:1789760746.944402:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736191232680 time 736252422913 state 0x2 ref 0 00000100:00100000:1.0:1789760746.951341:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760746.951442:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484246400 00000100:00100000:3.0:1789760746.951451:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484246400 00000100:00100000:3.0:1789760746.951509:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736191232680 time 736252422913 state 0x0 ref 1 00000100:00100000:3.0:1789760746.951752:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736252683329 time 736252422913 state 0x1 ref 1 00000100:00100000:3.0:1789760746.951784:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736252683329 time 736282715213 state 0x0 ref 1 00000100:00100000:3.0:1789760746.951808:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484246400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760746.951853:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21790720, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:3.0:1789760746.951862:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.951879:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:3.0:1789760746.951883:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760746.951890:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21790720->21794815] (req 21790720->21794815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760746.951905:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21790720->21794815] (req 21790720->21794815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760747.009812:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972638 is committed 00002000:00010000:1.0:1789760747.009925:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21790720 -> 21794816 00010000:00010000:1.0:1789760747.009941:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21790720->21794815] (req 21790720->21794815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.009955:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21790720->21794815] (req 21790720->21794815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.009964:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21790720->21794815] (req 21790720->21794815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.009972:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760747.009977:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21790720->21794815] (req 21790720->21794815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.009987:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21790720->21794815] (req 21790720->21794815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.010006:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21790720->21794815] (req 21790720->21794815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.010014:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208b35 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21790720->21794815] (req 21790720->21794815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760747.010423:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484246400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58612us (59085us total) trans 4294972638 rc 0/0 00000100:00100000:1.0:1789760747.010436:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5349 00000100:00100000:1.0:1789760747.010448:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736252683329 time 736282715213 state 0x2 ref 0 00000100:00100000:0.0:1789760747.022282:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760747.022687:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484246528 00000100:00100000:2.0:1789760747.022698:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484246528 00000100:00100000:2.0:1789760747.022751:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736252683329 time 736282715213 state 0x0 ref 1 00000100:00100000:2.0:1789760747.022767:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736282975629 time 736282715213 state 0x1 ref 1 00000100:00100000:2.0:1789760747.022807:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736282975629 time 736353738367 state 0x0 ref 1 00000100:00100000:2.0:1789760747.022823:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484246528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760747.022871:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21794816, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:2.0:1789760747.022880:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.022895:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:2.0:1789760747.022900:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760747.022907:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21794816->21798911] (req 21794816->21798911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.022921:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21794816->21798911] (req 21794816->21798911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760747.087981:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972639 is committed 00002000:00010000:1.0:1789760747.088245:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21794816 -> 21798912 00010000:00010000:1.0:1789760747.088262:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21794816->21798911] (req 21794816->21798911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.088354:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21794816->21798911] (req 21794816->21798911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.088363:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21794816->21798911] (req 21794816->21798911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.088369:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760747.088373:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21794816->21798911] (req 21794816->21798911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.088381:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21794816->21798911] (req 21794816->21798911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.088400:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21794816->21798911] (req 21794816->21798911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.088407:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208b3c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21794816->21798911] (req 21794816->21798911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760747.088521:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484246528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65696us (66253us total) trans 4294972639 rc 0/0 00000100:00100000:1.0:1789760747.088531:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5350 00000100:00100000:1.0:1789760747.088540:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736282975629 time 736353738367 state 0x2 ref 0 00000100:00100000:2.0:1789760747.095414:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760747.095570:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484246656 00000100:00100000:0.0:1789760747.095580:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484246656 00000100:00100000:0.0:1789760747.095682:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736282975629 time 736353738367 state 0x0 ref 1 00000100:00100000:0.0:1789760747.095696:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736353998783 time 736353738367 state 0x1 ref 1 00000100:00100000:0.0:1789760747.095726:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736353998783 time 736426656856 state 0x0 ref 1 00000100:00100000:0.0:1789760747.095740:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484246656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760747.095791:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21798912, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:0.0:1789760747.095801:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.095817:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:0.0:1789760747.095821:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760747.095827:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21798912->21803007] (req 21798912->21803007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.095841:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21798912->21803007] (req 21798912->21803007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760747.120695:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972640 is committed 00002000:00010000:1.0:1789760747.120825:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21798912 -> 21803008 00010000:00010000:1.0:1789760747.120840:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21798912->21803007] (req 21798912->21803007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.120853:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21798912->21803007] (req 21798912->21803007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.120861:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21798912->21803007] (req 21798912->21803007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.120868:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760747.120872:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21798912->21803007] (req 21798912->21803007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.120881:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21798912->21803007] (req 21798912->21803007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.120900:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21798912->21803007] (req 21798912->21803007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.120908:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b43 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21798912->21803007] (req 21798912->21803007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760747.121004:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484246656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25262us (25594us total) trans 4294972640 rc 0/0 00000100:00100000:1.0:1789760747.121013:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5351 00000100:00100000:1.0:1789760747.121024:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736353998783 time 736426656856 state 0x2 ref 0 00000100:00100000:0.0:1789760747.124810:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760747.124896:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484246784 00000100:00100000:1.0:1789760747.124905:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484246784 00000100:00100000:1.0:1789760747.124958:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736353998783 time 736426656856 state 0x0 ref 1 00000100:00100000:1.0:1789760747.124974:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736426917272 time 736426656856 state 0x1 ref 1 00000100:00100000:1.0:1789760747.125005:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736426917272 time 736455935569 state 0x0 ref 1 00000100:00100000:1.0:1789760747.125017:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484246784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760747.125110:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21803008, mtime: 0x6aad94ea, atime: 0x0, ctime: 0x6aad94ea, blocks: 0xa807 00010000:00010000:1.0:1789760747.125120:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.125135:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:1.0:1789760747.125139:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760747.125146:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21803008->21807103] (req 21803008->21807103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.125159:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21803008->21807103] (req 21803008->21807103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760747.160952:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972641 is committed 00002000:00010000:0.0:1789760747.161149:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21803008 -> 21807104 00002000:00010000:0.0:1789760747.161154:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760746 -> 1789760747 00002000:00010000:0.0:1789760747.161157:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760746 -> 1789760747 00010000:00010000:0.0:1789760747.161170:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21803008->21807103] (req 21803008->21807103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.161293:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21803008->21807103] (req 21803008->21807103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.161307:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21803008->21807103] (req 21803008->21807103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.161315:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760747.161320:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21803008->21807103] (req 21803008->21807103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.161331:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21803008->21807103] (req 21803008->21807103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.161351:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21803008->21807103] (req 21803008->21807103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.161358:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208b4a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21803008->21807103] (req 21803008->21807103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760747.161520:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484246784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36500us (36716us total) trans 4294972641 rc 0/0 00000100:00100000:0.0:1789760747.161530:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5352 00000100:00100000:0.0:1789760747.161541:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736426917272 time 736455935569 state 0x2 ref 0 00000100:00100000:0.0:1789760747.165999:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760747.166583:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484246912 00000100:00100000:1.0:1789760747.166594:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484246912 00000100:00100000:1.0:1789760747.166649:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736426917272 time 736455935569 state 0x0 ref 1 00000100:00100000:1.0:1789760747.166664:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736456195985 time 736455935569 state 0x1 ref 1 00000100:00100000:1.0:1789760747.166726:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736456195985 time 736497655969 state 0x0 ref 1 00000100:00100000:1.0:1789760747.166740:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484246912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760747.166785:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21807104, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:1.0:1789760747.166795:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.166810:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:1.0:1789760747.166815:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760747.166820:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21807104->21811199] (req 21807104->21811199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.166833:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21807104->21811199] (req 21807104->21811199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760747.208135:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972642 is committed 00002000:00010000:1.0:1789760747.210707:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21807104 -> 21811200 00010000:00010000:1.0:1789760747.210727:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21807104->21811199] (req 21807104->21811199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.210739:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21807104->21811199] (req 21807104->21811199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.210748:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21807104->21811199] (req 21807104->21811199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.210754:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760747.210760:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21807104->21811199] (req 21807104->21811199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.210769:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21807104->21811199] (req 21807104->21811199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.210787:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21807104->21811199] (req 21807104->21811199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.210794:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b51 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21807104->21811199] (req 21807104->21811199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760747.210944:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484246912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44202us (44951us total) trans 4294972642 rc 0/0 00000100:00100000:1.0:1789760747.210956:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5353 00000100:00100000:1.0:1789760747.210966:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736456195985 time 736497655969 state 0x2 ref 0 00000100:00100000:2.0:1789760747.217175:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760747.217328:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484247040 00000100:00100000:0.0:1789760747.217338:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484247040 00000100:00100000:0.0:1789760747.217393:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736456195985 time 736497655969 state 0x0 ref 1 00000100:00100000:0.0:1789760747.217408:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736497916385 time 736497655969 state 0x1 ref 1 00000100:00100000:0.0:1789760747.217434:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736497916385 time 736548366094 state 0x0 ref 1 00000100:00100000:0.0:1789760747.217446:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484247040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760747.217491:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21811200, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:0.0:1789760747.217501:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.217517:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:0.0:1789760747.217521:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760747.217528:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21811200->21815295] (req 21811200->21815295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.217541:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21811200->21815295] (req 21811200->21815295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760747.258467:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972643 is committed 00002000:00010000:0.0:1789760747.258744:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21811200 -> 21815296 00010000:00010000:0.0:1789760747.258759:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21811200->21815295] (req 21811200->21815295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.258771:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21811200->21815295] (req 21811200->21815295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.258781:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21811200->21815295] (req 21811200->21815295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.258788:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760747.258793:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21811200->21815295] (req 21811200->21815295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.258801:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21811200->21815295] (req 21811200->21815295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.258826:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21811200->21815295] (req 21811200->21815295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.258834:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208b58 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21811200->21815295] (req 21811200->21815295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760747.258929:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484247040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41480us (41760us total) trans 4294972643 rc 0/0 00000100:00100000:0.0:1789760747.258939:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5354 00000100:00100000:0.0:1789760747.258948:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736497916385 time 736548366094 state 0x2 ref 0 00000100:00100000:0.0:1789760747.264486:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760747.264742:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484247168 00000100:00100000:1.0:1789760747.264751:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484247168 00000100:00100000:1.0:1789760747.264805:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736497916385 time 736548366094 state 0x0 ref 1 00000100:00100000:1.0:1789760747.264819:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736548626510 time 736548366094 state 0x1 ref 1 00000100:00100000:1.0:1789760747.264850:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736548626510 time 736595781041 state 0x0 ref 1 00000100:00100000:1.0:1789760747.264861:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484247168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760747.264914:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21815296, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:1.0:1789760747.264924:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.264939:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:1.0:1789760747.264943:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760747.264949:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21815296->21819391] (req 21815296->21819391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.264962:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21815296->21819391] (req 21815296->21819391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760747.303979:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21815296 -> 21819392 00010000:00010000:1.0:1789760747.303999:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21815296->21819391] (req 21815296->21819391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.304016:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21815296->21819391] (req 21815296->21819391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.304025:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21815296->21819391] (req 21815296->21819391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.304032:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760747.304039:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21815296->21819391] (req 21815296->21819391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.304047:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21815296->21819391] (req 21815296->21819391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.304096:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21815296->21819391] (req 21815296->21819391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.304104:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b5f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21815296->21819391] (req 21815296->21819391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760747.304141:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972644 is committed 00000100:00100000:1.0:1789760747.304199:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484247168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39334us (39717us total) trans 4294972644 rc 0/0 00000100:00100000:1.0:1789760747.304209:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5355 00000100:00100000:1.0:1789760747.304218:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736548626510 time 736595781041 state 0x2 ref 0 00000100:00100000:2.0:1789760747.308860:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760747.308958:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484247296 00000100:00100000:1.0:1789760747.308967:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484247296 00000100:00100000:1.0:1789760747.309020:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736548626510 time 736595781041 state 0x0 ref 1 00000100:00100000:1.0:1789760747.309035:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736596041457 time 736595781041 state 0x1 ref 1 00000100:00100000:1.0:1789760747.309378:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736596041457 time 736640307665 state 0x0 ref 1 00000100:00100000:1.0:1789760747.309394:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484247296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760747.309444:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21819392, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:1.0:1789760747.309453:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.309469:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:1.0:1789760747.309474:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760747.309480:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21819392->21823487] (req 21819392->21823487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.309492:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21819392->21823487] (req 21819392->21823487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760747.342014:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21819392 -> 21823488 00010000:00010000:1.0:1789760747.342034:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21819392->21823487] (req 21819392->21823487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.342047:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21819392->21823487] (req 21819392->21823487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.342087:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21819392->21823487] (req 21819392->21823487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.342095:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760747.342101:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21819392->21823487] (req 21819392->21823487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.342110:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21819392->21823487] (req 21819392->21823487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.342128:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21819392->21823487] (req 21819392->21823487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.342135:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b66 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21819392->21823487] (req 21819392->21823487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760747.342234:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484247296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32838us (33377us total) trans 4294972645 rc 0/0 00000100:00100000:1.0:1789760747.342245:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5356 00000100:00100000:1.0:1789760747.342255:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736596041457 time 736640307665 state 0x2 ref 0 00000001:00080000:0.0:1789760747.342934:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972645 is committed 00000100:00100000:0.0:1789760747.345438:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760747.345526:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484247424 00000100:00100000:3.0:1789760747.345536:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484247424 00000100:00100000:3.0:1789760747.345589:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736596041457 time 736640307665 state 0x0 ref 1 00000100:00100000:3.0:1789760747.345604:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736640568081 time 736640307665 state 0x1 ref 1 00000100:00100000:3.0:1789760747.345634:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736640568081 time 736676565194 state 0x0 ref 1 00000100:00100000:3.0:1789760747.345650:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484247424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760747.345812:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21823488, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:3.0:1789760747.345822:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.345839:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:3.0:1789760747.345843:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760747.345849:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21823488->21827583] (req 21823488->21827583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.345863:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21823488->21827583] (req 21823488->21827583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760747.377110:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972646 is committed 00002000:00010000:0.0:1789760747.377222:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21823488 -> 21827584 00010000:00010000:0.0:1789760747.377238:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21823488->21827583] (req 21823488->21827583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.377252:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21823488->21827583] (req 21823488->21827583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.377262:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21823488->21827583] (req 21823488->21827583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.377281:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760747.377287:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21823488->21827583] (req 21823488->21827583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.377296:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21823488->21827583] (req 21823488->21827583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.377317:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21823488->21827583] (req 21823488->21827583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.377325:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208b6d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21823488->21827583] (req 21823488->21827583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760747.377425:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484247424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31772us (31991us total) trans 4294972646 rc 0/0 00000100:00100000:0.0:1789760747.377436:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5357 00000100:00100000:0.0:1789760747.377446:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736640568081 time 736676565194 state 0x2 ref 0 00000100:00100000:3.0:1789760747.381233:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760747.381344:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484247552 00000100:00100000:0.0:1789760747.381353:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484247552 00000100:00100000:0.0:1789760747.381420:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736640568081 time 736676565194 state 0x0 ref 1 00000100:00100000:0.0:1789760747.381435:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736676825610 time 736676565194 state 0x1 ref 1 00000100:00100000:0.0:1789760747.381458:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736676825610 time 736712388546 state 0x0 ref 1 00000100:00100000:0.0:1789760747.381470:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484247552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760747.381518:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21827584, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:0.0:1789760747.381529:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.381545:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:0.0:1789760747.381550:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760747.381556:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21827584->21831679] (req 21827584->21831679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.381568:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21827584->21831679] (req 21827584->21831679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.405981:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:3.0:1789760747.417957:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21827584 -> 21831680 00010000:00010000:3.0:1789760747.417977:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21827584->21831679] (req 21827584->21831679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.417992:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21827584->21831679] (req 21827584->21831679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.418002:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21827584->21831679] (req 21827584->21831679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.418009:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760747.418015:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21827584->21831679] (req 21827584->21831679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.418025:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21827584->21831679] (req 21827584->21831679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.418045:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21827584->21831679] (req 21827584->21831679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.418091:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208b74 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21827584->21831679] (req 21827584->21831679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760747.418182:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484247552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36709us (36952us total) trans 4294972647 rc 0/0 00000100:00100000:3.0:1789760747.418193:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5358 00000100:00100000:3.0:1789760747.418203:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736676825610 time 736712388546 state 0x2 ref 0 00000001:00080000:3.0:1789760747.418241:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972647 is committed 00000100:00100000:3.0:1789760747.425017:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760747.425118:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484247680 00000100:00100000:1.0:1789760747.425128:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484247680 00000100:00100000:1.0:1789760747.425181:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736676825610 time 736712388546 state 0x0 ref 1 00000100:00100000:1.0:1789760747.425196:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736712648962 time 736712388546 state 0x1 ref 1 00000100:00100000:1.0:1789760747.425224:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736712648962 time 736756155292 state 0x0 ref 1 00000100:00100000:1.0:1789760747.425236:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484247680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760747.425309:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21831680, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:1.0:1789760747.425326:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.425343:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:1.0:1789760747.425347:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760747.425354:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21831680->21835775] (req 21831680->21835775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.425379:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21831680->21835775] (req 21831680->21835775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760747.483572:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972648 is committed 00002000:00010000:1.0:1789760747.483744:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21831680 -> 21835776 00010000:00010000:1.0:1789760747.483762:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21831680->21835775] (req 21831680->21835775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.483775:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21831680->21835775] (req 21831680->21835775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.483784:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21831680->21835775] (req 21831680->21835775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.483791:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760747.483796:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21831680->21835775] (req 21831680->21835775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.483805:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21831680->21835775] (req 21831680->21835775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.483827:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21831680->21835775] (req 21831680->21835775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.483834:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b7b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21831680->21835775] (req 21831680->21835775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760747.483966:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484247680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58727us (58955us total) trans 4294972648 rc 0/0 00000100:00100000:1.0:1789760747.483979:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5359 00000100:00100000:1.0:1789760747.483987:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736712648962 time 736756155292 state 0x2 ref 0 00000100:00100000:0.0:1789760747.489577:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760747.489945:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484247808 00000100:00100000:1.0:1789760747.489955:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484247808 00000100:00100000:1.0:1789760747.490009:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736712648962 time 736756155292 state 0x0 ref 1 00000100:00100000:1.0:1789760747.490025:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736756415708 time 736756155292 state 0x1 ref 1 00000100:00100000:1.0:1789760747.490193:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736756415708 time 736821122735 state 0x0 ref 1 00000100:00100000:1.0:1789760747.490216:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484247808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760747.490265:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21835776, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:1.0:1789760747.490442:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.490459:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:1.0:1789760747.490464:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760747.490471:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21835776->21839871] (req 21835776->21839871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.490486:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21835776->21839871] (req 21835776->21839871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760747.526632:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972649 is committed 00002000:00010000:1.0:1789760747.526905:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21835776 -> 21839872 00010000:00010000:1.0:1789760747.526921:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21835776->21839871] (req 21835776->21839871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.526934:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21835776->21839871] (req 21835776->21839871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.526944:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21835776->21839871] (req 21835776->21839871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.526951:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760747.526956:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21835776->21839871] (req 21835776->21839871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.526965:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21835776->21839871] (req 21835776->21839871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.526985:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21835776->21839871] (req 21835776->21839871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.526994:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208b82 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21835776->21839871] (req 21835776->21839871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760747.527128:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484247808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36911us (38075us total) trans 4294972649 rc 0/0 00000100:00100000:1.0:1789760747.527141:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5360 00000100:00100000:1.0:1789760747.527150:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736756415708 time 736821122735 state 0x2 ref 0 00000100:00100000:3.0:1789760747.532562:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760747.532642:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484247936 00000100:00100000:0.0:1789760747.532652:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484247936 00000100:00100000:0.0:1789760747.532709:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736756415708 time 736821122735 state 0x0 ref 1 00000100:00100000:0.0:1789760747.532724:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736821383151 time 736821122735 state 0x1 ref 1 00000100:00100000:0.0:1789760747.532754:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736821383151 time 736863685498 state 0x0 ref 1 00000100:00100000:0.0:1789760747.532769:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484247936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760747.532821:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21839872, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:0.0:1789760747.532830:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.532846:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:0.0:1789760747.532851:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760747.532858:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21839872->21843967] (req 21839872->21843967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.532873:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21839872->21843967] (req 21839872->21843967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760747.571667:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21839872 -> 21843968 00010000:00010000:3.0:1789760747.571686:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21839872->21843967] (req 21839872->21843967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.571699:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21839872->21843967] (req 21839872->21843967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.571712:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21839872->21843967] (req 21839872->21843967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.571719:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760747.571723:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21839872->21843967] (req 21839872->21843967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.571732:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21839872->21843967] (req 21839872->21843967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.571754:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21839872->21843967] (req 21839872->21843967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.571762:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208b89 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21839872->21843967] (req 21839872->21843967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760747.571867:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484247936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39096us (39310us total) trans 4294972650 rc 0/0 00000100:00100000:3.0:1789760747.571877:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5361 00000100:00100000:3.0:1789760747.571887:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736821383151 time 736863685498 state 0x2 ref 0 00000100:00100000:3.0:1789760747.574834:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760747.574989:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484248064 00000100:00100000:0.0:1789760747.574999:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484248064 00000100:00100000:0.0:1789760747.575093:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736821383151 time 736863685498 state 0x0 ref 1 00000100:00100000:0.0:1789760747.575112:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736863945914 time 736863685498 state 0x1 ref 1 00000100:00100000:0.0:1789760747.575133:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736863945914 time 736906064519 state 0x0 ref 1 00000100:00100000:0.0:1789760747.575146:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484248064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760747.575193:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21843968, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:0.0:1789760747.575203:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.575217:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:0.0:1789760747.575221:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760747.575227:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21843968->21848063] (req 21843968->21848063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.575241:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21843968->21848063] (req 21843968->21848063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760747.579799:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972650 is committed 00000001:00080000:3.0:1789760747.611616:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972651 is committed 00002000:00010000:2.0:1789760747.611819:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21843968 -> 21848064 00010000:00010000:2.0:1789760747.611834:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21843968->21848063] (req 21843968->21848063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.611848:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21843968->21848063] (req 21843968->21848063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.611863:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21843968->21848063] (req 21843968->21848063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.611870:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760747.611875:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21843968->21848063] (req 21843968->21848063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.611884:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21843968->21848063] (req 21843968->21848063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.611903:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21843968->21848063] (req 21843968->21848063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.611910:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208b90 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21843968->21848063] (req 21843968->21848063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760747.612039:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484248064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36891us (37209us total) trans 4294972651 rc 0/0 00000100:00100000:2.0:1789760747.612059:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5362 00000100:00100000:2.0:1789760747.612109:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736863945914 time 736906064519 state 0x2 ref 0 00000100:00100000:1.0:1789760747.616253:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760747.616557:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484248192 00000100:00100000:2.0:1789760747.616567:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484248192 00000100:00100000:2.0:1789760747.616624:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736863945914 time 736906064519 state 0x0 ref 1 00000100:00100000:2.0:1789760747.616638:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736906324935 time 736906064519 state 0x1 ref 1 00000100:00100000:2.0:1789760747.616708:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736906324935 time 736947638168 state 0x0 ref 1 00000100:00100000:2.0:1789760747.616720:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484248192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760747.616771:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21848064, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:2.0:1789760747.616779:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.616794:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:2.0:1789760747.616798:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760747.616804:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21848064->21852159] (req 21848064->21852159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.616816:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21848064->21852159] (req 21848064->21852159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760747.650640:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972652 is committed 00002000:00010000:2.0:1789760747.650743:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21848064 -> 21852160 00010000:00010000:2.0:1789760747.650758:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21848064->21852159] (req 21848064->21852159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.650771:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21848064->21852159] (req 21848064->21852159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.650781:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21848064->21852159] (req 21848064->21852159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.650789:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760747.650795:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21848064->21852159] (req 21848064->21852159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.650804:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21848064->21852159] (req 21848064->21852159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.650824:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21848064->21852159] (req 21848064->21852159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.650832:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208b97 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21848064->21852159] (req 21848064->21852159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760747.650919:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484248192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34196us (34671us total) trans 4294972652 rc 0/0 00000100:00100000:2.0:1789760747.650929:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5363 00000100:00100000:2.0:1789760747.650940:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736906324935 time 736947638168 state 0x2 ref 0 00000100:00100000:0.0:1789760747.655881:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760747.655992:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484248320 00000100:00100000:1.0:1789760747.656009:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484248320 00000100:00100000:1.0:1789760747.656184:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736906324935 time 736947638168 state 0x0 ref 1 00000100:00100000:1.0:1789760747.656201:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736947898584 time 736947638168 state 0x1 ref 1 00000100:00100000:1.0:1789760747.656292:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736947898584 time 736987222017 state 0x0 ref 1 00000100:00100000:1.0:1789760747.656305:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484248320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760747.656351:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21852160, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:1.0:1789760747.656727:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.656745:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=736s v=(0 0) 00010000:00010000:1.0:1789760747.656749:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760747.656755:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21852160->21856255] (req 21852160->21856255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.656768:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21852160->21856255] (req 21852160->21856255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760747.708091:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972653 is committed 00002000:00010000:0.0:1789760747.708242:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21852160 -> 21856256 00010000:00010000:0.0:1789760747.708257:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21852160->21856255] (req 21852160->21856255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.708340:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21852160->21856255] (req 21852160->21856255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.708352:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21852160->21856255] (req 21852160->21856255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.708472:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760747.708478:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21852160->21856255] (req 21852160->21856255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.708486:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21852160->21856255] (req 21852160->21856255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.708508:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21852160->21856255] (req 21852160->21856255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.708515:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208b9e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21852160->21856255] (req 21852160->21856255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760747.708637:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484248320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52331us (52760us total) trans 4294972653 rc 0/0 00000100:00100000:0.0:1789760747.708648:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5364 00000100:00100000:0.0:1789760747.708659:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736947898584 time 736987222017 state 0x2 ref 0 00000100:00100000:0.0:1789760747.712147:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760747.712236:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484248448 00000100:00100000:3.0:1789760747.712245:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484248448 00000100:00100000:3.0:1789760747.712417:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736947898584 time 736987222017 state 0x0 ref 1 00000100:00100000:3.0:1789760747.712433:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736987482433 time 736987222017 state 0x1 ref 1 00000100:00100000:3.0:1789760747.712495:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736987482433 time 737043425082 state 0x0 ref 1 00000100:00100000:3.0:1789760747.712507:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484248448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760747.712556:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21856256, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:3.0:1789760747.712565:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.712579:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:3.0:1789760747.712584:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760747.712591:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21856256->21860351] (req 21856256->21860351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760747.712605:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21856256->21860351] (req 21856256->21860351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760747.744646:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972654 is committed 00002000:00010000:2.0:1789760747.744763:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21856256 -> 21860352 00010000:00010000:2.0:1789760747.744780:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21856256->21860351] (req 21856256->21860351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.744793:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21856256->21860351] (req 21856256->21860351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.744804:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21856256->21860351] (req 21856256->21860351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.744818:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760747.744823:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21856256->21860351] (req 21856256->21860351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.744832:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21856256->21860351] (req 21856256->21860351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.744852:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21856256->21860351] (req 21856256->21860351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.744861:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208ba5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21856256->21860351] (req 21856256->21860351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760747.744955:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484248448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32444us (32812us total) trans 4294972654 rc 0/0 00000100:00100000:2.0:1789760747.744965:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5365 00000100:00100000:2.0:1789760747.744975:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736987482433 time 737043425082 state 0x2 ref 0 00000100:00100000:0.0:1789760747.749110:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760747.749143:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760747.749181:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760747.749224:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760747.749250:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484248832 00000100:00100000:0.0:1789760747.749260:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484248832 00000100:00100000:3.0:1789760747.749371:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484249088 00000100:00100000:3.0:1789760747.749375:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484249088 00000100:00100000:3.0:1789760747.749433:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 931 00000100:00100000:3.0:1789760747.749444:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484249088:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760747.749525:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484249088:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 83us (343us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760747.749536:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 931 00000100:00080000:3.0:1789760747.749559:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:0.0:1789760747.749699:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 932 00000100:00100000:0.0:1789760747.749708:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484248832:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760747.749752:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484248832:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 43us (605us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760747.749760:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 932 00000100:00100000:0.0:1789760747.749781:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484248704 00000100:00100000:0.0:1789760747.749784:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484248704 00000100:00100000:0.0:1789760747.749819:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 481 00000100:00100000:0.0:1789760747.749834:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b7c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484248704:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760747.749863:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b7c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484248704:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 37us (638us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760747.749869:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 481 00000100:00080000:3.0:1789760747.749979:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:2.0:1789760747.750055:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484248576 00000100:00100000:2.0:1789760747.750060:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484248576 00000100:00100000:2.0:1789760747.750114:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 736987482433 time 737043425082 state 0x0 ref 1 00000100:00100000:2.0:1789760747.750128:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737043685498 time 737043425082 state 0x1 ref 1 00000100:00100000:2.0:1789760747.750152:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737043685498 time 737081083438 state 0x0 ref 1 00000100:00100000:2.0:1789760747.750166:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484248576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760747.750209:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21860352, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:2.0:1789760747.750218:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.750233:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:2.0:1789760747.750238:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760747.750244:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21860352->21864447] (req 21860352->21864447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.750259:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21860352->21864447] (req 21860352->21864447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760747.755414:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760747.755466:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760747.756171:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484248960 00000100:00100000:3.0:1789760747.756181:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484248960 00000100:00100000:3.0:1789760747.756233:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484249216 00000100:00100000:3.0:1789760747.756237:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484249216 00000100:00100000:3.0:1789760747.756286:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 933 00000100:00100000:3.0:1789760747.756298:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484248960:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760747.756373:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484248960:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 77us (968us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760747.756381:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 933 00000100:00100000:3.0:1789760747.756398:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 934 00000100:00100000:3.0:1789760747.756405:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb5c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484249216:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760747.756432:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb5c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484249216:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 27us (965us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760747.756438:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 934 00000100:00080000:1.0:1789760747.756628:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000001:00080000:0.0:1789760747.779843:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972655 is committed 00002000:00010000:2.0:1789760747.779965:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21860352 -> 21864448 00010000:00010000:2.0:1789760747.779979:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21860352->21864447] (req 21860352->21864447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.779992:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21860352->21864447] (req 21860352->21864447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.780000:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21860352->21864447] (req 21860352->21864447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.780008:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760747.780013:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21860352->21864447] (req 21860352->21864447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.780022:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21860352->21864447] (req 21860352->21864447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.780041:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21860352->21864447] (req 21860352->21864447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.780209:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bac lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21860352->21864447] (req 21860352->21864447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760747.780453:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484248576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30284us (31348us total) trans 4294972655 rc 0/0 00000100:00100000:2.0:1789760747.780464:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5366 00000100:00100000:2.0:1789760747.780475:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737043685498 time 737081083438 state 0x2 ref 0 00000100:00100000:1.0:1789760747.795402:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760747.795509:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484249344 00000100:00100000:2.0:1789760747.795518:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484249344 00000100:00100000:2.0:1789760747.795572:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737043685498 time 737081083438 state 0x0 ref 1 00000100:00100000:2.0:1789760747.795587:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737081343854 time 737081083438 state 0x1 ref 1 00000100:00100000:2.0:1789760747.795629:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737081343854 time 737126559249 state 0x0 ref 1 00000100:00100000:2.0:1789760747.795642:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484249344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760747.795689:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21864448, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:2.0:1789760747.795699:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.795715:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:2.0:1789760747.795720:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760747.795726:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21864448->21868543] (req 21864448->21868543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.795739:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21864448->21868543] (req 21864448->21868543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760747.828639:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972656 is committed 00002000:00010000:2.0:1789760747.828673:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21864448 -> 21868544 00010000:00010000:2.0:1789760747.828690:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21864448->21868543] (req 21864448->21868543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.828703:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21864448->21868543] (req 21864448->21868543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.828712:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21864448->21868543] (req 21864448->21868543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.828720:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760747.828724:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21864448->21868543] (req 21864448->21868543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.828734:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21864448->21868543] (req 21864448->21868543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.828752:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21864448->21868543] (req 21864448->21868543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.828761:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bb3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21864448->21868543] (req 21864448->21868543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760747.828846:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484249344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33203us (33449us total) trans 4294972656 rc 0/0 00000100:00100000:2.0:1789760747.828856:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5367 00000100:00100000:2.0:1789760747.828866:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737081343854 time 737126559249 state 0x2 ref 0 00000100:00100000:0.0:1789760747.836361:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760747.836451:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484249472 00000100:00100000:1.0:1789760747.836461:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484249472 00000100:00100000:1.0:1789760747.836514:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737081343854 time 737126559249 state 0x0 ref 1 00000100:00100000:1.0:1789760747.836530:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737126819665 time 737126559249 state 0x1 ref 1 00000100:00100000:1.0:1789760747.836555:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737126819665 time 737167486749 state 0x0 ref 1 00000100:00100000:1.0:1789760747.836568:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484249472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760747.836616:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21868544, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:1.0:1789760747.836626:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.836642:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:1.0:1789760747.836646:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760747.836653:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21868544->21872639] (req 21868544->21872639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760747.836678:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21868544->21872639] (req 21868544->21872639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760747.911060:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21868544 -> 21872640 00010000:00010000:0.0:1789760747.911092:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21868544->21872639] (req 21868544->21872639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.911101:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21868544->21872639] (req 21868544->21872639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.911107:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21868544->21872639] (req 21868544->21872639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.911112:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760747.911117:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21868544->21872639] (req 21868544->21872639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.911126:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21868544->21872639] (req 21868544->21872639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.911145:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21868544->21872639] (req 21868544->21872639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.911152:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208bba lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21868544->21872639] (req 21868544->21872639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760747.911242:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484249472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 74671us (74884us total) trans 4294972657 rc 0/0 00000100:00100000:0.0:1789760747.911253:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5368 00000100:00100000:0.0:1789760747.911261:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737126819665 time 737167486749 state 0x2 ref 0 00000001:00080000:3.0:1789760747.911348:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972657 is committed 00000100:00100000:3.0:1789760747.915455:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760747.915553:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484249600 00000100:00100000:2.0:1789760747.915571:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484249600 00000100:00100000:2.0:1789760747.915628:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737126819665 time 737167486749 state 0x0 ref 1 00000100:00100000:2.0:1789760747.915642:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737167747165 time 737167486749 state 0x1 ref 1 00000100:00100000:2.0:1789760747.915671:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737167747165 time 737246602819 state 0x0 ref 1 00000100:00100000:2.0:1789760747.915683:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484249600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760747.915725:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21872640, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:2.0:1789760747.915733:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.915749:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:2.0:1789760747.915753:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760747.915759:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21872640->21876735] (req 21872640->21876735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.915777:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21872640->21876735] (req 21872640->21876735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760747.965234:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21872640 -> 21876736 00010000:00010000:0.0:1789760747.965253:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21872640->21876735] (req 21872640->21876735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.965279:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21872640->21876735] (req 21872640->21876735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.965290:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21872640->21876735] (req 21872640->21876735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.965297:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760747.965302:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21872640->21876735] (req 21872640->21876735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.965312:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21872640->21876735] (req 21872640->21876735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.965331:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21872640->21876735] (req 21872640->21876735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760747.965338:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208bc1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21872640->21876735] (req 21872640->21876735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760747.965438:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484249600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49752us (49986us total) trans 4294972658 rc 0/0 00000100:00100000:0.0:1789760747.965449:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5369 00000100:00100000:0.0:1789760747.965460:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737167747165 time 737246602819 state 0x2 ref 0 00000001:00080000:1.0:1789760747.966526:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972658 is committed 00000100:00100000:1.0:1789760747.968778:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760747.969345:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484249728 00000100:00100000:2.0:1789760747.969421:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484249728 00000100:00100000:2.0:1789760747.969481:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737167747165 time 737246602819 state 0x0 ref 1 00000100:00100000:2.0:1789760747.969497:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737246863235 time 737246602819 state 0x1 ref 1 00000100:00100000:2.0:1789760747.969558:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737246863235 time 737300488551 state 0x0 ref 1 00000100:00100000:2.0:1789760747.969571:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484249728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760747.969626:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21876736, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:2.0:1789760747.969634:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.969648:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:2.0:1789760747.969653:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760747.969659:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21876736->21880831] (req 21876736->21880831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760747.969673:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21876736->21880831] (req 21876736->21880831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760748.004488:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21876736 -> 21880832 00010000:00010000:3.0:1789760748.004508:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21876736->21880831] (req 21876736->21880831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.004522:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21876736->21880831] (req 21876736->21880831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.004537:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21876736->21880831] (req 21876736->21880831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.004544:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760748.004549:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21876736->21880831] (req 21876736->21880831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.004557:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21876736->21880831] (req 21876736->21880831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.004588:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21876736->21880831] (req 21876736->21880831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.004595:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208bc8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21876736->21880831] (req 21876736->21880831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760748.004688:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484249728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35115us (35914us total) trans 4294972659 rc 0/0 00000100:00100000:3.0:1789760748.004699:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5370 00000100:00100000:3.0:1789760748.004709:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737246863235 time 737300488551 state 0x2 ref 0 00000001:00080000:0.0:1789760748.005418:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972659 is committed 00000100:00100000:0.0:1789760748.010769:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760748.010863:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484249856 00000100:00100000:1.0:1789760748.010872:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484249856 00000100:00080000:3.0:1789760748.010916:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1789760748.010957:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737246863235 time 737300488551 state 0x0 ref 1 00000100:00100000:1.0:1789760748.010976:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737300748967 time 737300488551 state 0x1 ref 1 00000100:00100000:1.0:1789760748.011059:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737300748967 time 737341982402 state 0x0 ref 1 00000100:00100000:1.0:1789760748.011111:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484249856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760748.011161:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21880832, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:1.0:1789760748.011170:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.011187:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:1.0:1789760748.011192:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760748.011199:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21880832->21884927] (req 21880832->21884927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.011212:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21880832->21884927] (req 21880832->21884927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760748.039433:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21880832 -> 21884928 00010000:00010000:3.0:1789760748.039452:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21880832->21884927] (req 21880832->21884927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.039471:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21880832->21884927] (req 21880832->21884927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.039481:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21880832->21884927] (req 21880832->21884927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.039488:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760748.039493:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21880832->21884927] (req 21880832->21884927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.039501:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21880832->21884927] (req 21880832->21884927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.039522:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21880832->21884927] (req 21880832->21884927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.039529:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208bcf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21880832->21884927] (req 21880832->21884927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760748.039689:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484249856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28575us (28923us total) trans 4294972660 rc 0/0 00000100:00100000:3.0:1789760748.039701:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5371 00000100:00100000:3.0:1789760748.039710:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737300748967 time 737341982402 state 0x2 ref 0 00000001:00080000:0.0:1789760748.040244:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972660 is committed 00000100:00100000:1.0:1789760748.044359:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760748.044434:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484249984 00000100:00100000:0.0:1789760748.044444:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484249984 00000100:00100000:0.0:1789760748.044507:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737300748967 time 737341982402 state 0x0 ref 1 00000100:00100000:0.0:1789760748.044522:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737342242818 time 737341982402 state 0x1 ref 1 00000100:00100000:0.0:1789760748.044556:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737342242818 time 737375487478 state 0x0 ref 1 00000100:00100000:0.0:1789760748.044685:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484249984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760748.044735:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21884928, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:0.0:1789760748.044743:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.044758:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:0.0:1789760748.044762:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760748.044767:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21884928->21889023] (req 21884928->21889023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.044781:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21884928->21889023] (req 21884928->21889023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760748.073686:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21884928 -> 21889024 00010000:00010000:1.0:1789760748.073706:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21884928->21889023] (req 21884928->21889023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.073719:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21884928->21889023] (req 21884928->21889023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.073728:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21884928->21889023] (req 21884928->21889023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.073740:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760748.073745:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21884928->21889023] (req 21884928->21889023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.073755:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21884928->21889023] (req 21884928->21889023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.073775:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21884928->21889023] (req 21884928->21889023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.073783:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bd6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21884928->21889023] (req 21884928->21889023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760748.073881:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484249984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29196us (29525us total) trans 4294972661 rc 0/0 00000100:00100000:1.0:1789760748.073891:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5372 00000100:00100000:1.0:1789760748.073906:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737342242818 time 737375487478 state 0x2 ref 0 00000001:00080000:2.0:1789760748.074789:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972661 is committed 00000100:00100000:1.0:1789760748.077772:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760748.077866:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484250112 00000100:00100000:0.0:1789760748.077876:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484250112 00000100:00100000:0.0:1789760748.077937:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737342242818 time 737375487478 state 0x0 ref 1 00000100:00100000:0.0:1789760748.077953:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737375747894 time 737375487478 state 0x1 ref 1 00000100:00100000:0.0:1789760748.078023:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737375747894 time 737408953275 state 0x0 ref 1 00000100:00100000:0.0:1789760748.078036:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484250112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760748.078116:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21889024, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:0.0:1789760748.078125:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.078140:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:0.0:1789760748.078144:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760748.078150:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21889024->21893119] (req 21889024->21893119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.078163:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21889024->21893119] (req 21889024->21893119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760748.108028:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972662 is committed 00002000:00010000:2.0:1789760748.108243:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21889024 -> 21893120 00010000:00010000:2.0:1789760748.108258:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21889024->21893119] (req 21889024->21893119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.108281:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21889024->21893119] (req 21889024->21893119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.108289:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21889024->21893119] (req 21889024->21893119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.108296:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760748.108300:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21889024->21893119] (req 21889024->21893119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.108309:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21889024->21893119] (req 21889024->21893119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.108329:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21889024->21893119] (req 21889024->21893119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.108335:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396208bdd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21889024->21893119] (req 21889024->21893119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760748.108535:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484250112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30497us (30769us total) trans 4294972662 rc 0/0 00000100:00100000:2.0:1789760748.108545:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5373 00000100:00100000:2.0:1789760748.108554:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737375747894 time 737408953275 state 0x2 ref 0 00000100:00100000:1.0:1789760748.112870:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760748.112950:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484250240 00000100:00100000:3.0:1789760748.112959:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484250240 00000100:00100000:3.0:1789760748.113005:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737375747894 time 737408953275 state 0x0 ref 1 00000100:00100000:3.0:1789760748.113019:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737409213691 time 737408953275 state 0x1 ref 1 00000100:00100000:2.0:1789760748.113327:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737409213691 time 737444255801 state 0x0 ref 1 00000100:00100000:2.0:1789760748.113342:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484250240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760748.113400:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21893120, mtime: 0x6aad94eb, atime: 0x0, ctime: 0x6aad94eb, blocks: 0xa807 00010000:00010000:2.0:1789760748.113409:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.113425:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:2.0:1789760748.113429:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760748.113435:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21893120->21897215] (req 21893120->21897215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.113447:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21893120->21897215] (req 21893120->21897215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760748.137906:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972663 is committed 00002000:00010000:3.0:1789760748.138235:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21893120 -> 21897216 00002000:00010000:3.0:1789760748.138241:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760747 -> 1789760748 00002000:00010000:3.0:1789760748.138243:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760747 -> 1789760748 00010000:00010000:3.0:1789760748.138256:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21893120->21897215] (req 21893120->21897215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.138279:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21893120->21897215] (req 21893120->21897215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.138287:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21893120->21897215] (req 21893120->21897215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.138294:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760748.138299:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21893120->21897215] (req 21893120->21897215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.138307:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21893120->21897215] (req 21893120->21897215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.138327:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21893120->21897215] (req 21893120->21897215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.138340:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208be4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21893120->21897215] (req 21893120->21897215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760748.138493:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484250240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25150us (25626us total) trans 4294972663 rc 0/0 00000100:00100000:3.0:1789760748.138504:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5374 00000100:00100000:3.0:1789760748.138514:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737409213691 time 737444255801 state 0x2 ref 0 00000100:00100000:1.0:1789760748.142633:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760748.142708:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484250368 00000100:00100000:3.0:1789760748.142717:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484250368 00000100:00100000:3.0:1789760748.142772:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737409213691 time 737444255801 state 0x0 ref 1 00000100:00100000:3.0:1789760748.142788:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737444516217 time 737444255801 state 0x1 ref 1 00000100:00100000:3.0:1789760748.142819:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737444516217 time 737473749632 state 0x0 ref 1 00000100:00100000:3.0:1789760748.142833:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484250368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760748.142882:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21897216, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:3.0:1789760748.142891:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.142906:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:3.0:1789760748.142910:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760748.142917:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21897216->21901311] (req 21897216->21901311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.142936:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21897216->21901311] (req 21897216->21901311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760748.184592:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21897216 -> 21901312 00010000:00010000:1.0:1789760748.184612:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21897216->21901311] (req 21897216->21901311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.184637:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21897216->21901311] (req 21897216->21901311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.184646:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21897216->21901311] (req 21897216->21901311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.184653:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760748.184658:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21897216->21901311] (req 21897216->21901311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.184667:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21897216->21901311] (req 21897216->21901311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.184686:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21897216->21901311] (req 21897216->21901311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.184694:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208beb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21897216->21901311] (req 21897216->21901311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760748.184786:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484250368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41951us (42224us total) trans 4294972664 rc 0/0 00000100:00100000:1.0:1789760748.184798:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5375 00000100:00100000:1.0:1789760748.184807:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737444516217 time 737473749632 state 0x2 ref 0 00000001:00080000:3.0:1789760748.184898:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972664 is committed 00000100:00100000:1.0:1789760748.189602:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760748.189933:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484250496 00000100:00100000:0.0:1789760748.189942:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484250496 00000100:00100000:0.0:1789760748.190001:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737444516217 time 737473749632 state 0x0 ref 1 00000100:00100000:0.0:1789760748.190016:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737474010048 time 737473749632 state 0x1 ref 1 00000100:00100000:0.0:1789760748.190051:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737474010048 time 737520982709 state 0x0 ref 1 00000100:00100000:0.0:1789760748.190092:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484250496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760748.190141:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21901312, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:0.0:1789760748.190150:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.190167:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:0.0:1789760748.190172:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760748.190178:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21901312->21905407] (req 21901312->21905407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.190193:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21901312->21905407] (req 21901312->21905407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760748.235189:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972665 is committed 00002000:00010000:1.0:1789760748.235301:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21901312 -> 21905408 00010000:00010000:1.0:1789760748.235316:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21901312->21905407] (req 21901312->21905407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.235330:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21901312->21905407] (req 21901312->21905407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.235339:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21901312->21905407] (req 21901312->21905407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.235345:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760748.235350:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21901312->21905407] (req 21901312->21905407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.235358:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21901312->21905407] (req 21901312->21905407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.235376:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21901312->21905407] (req 21901312->21905407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.235384:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208bf2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21901312->21905407] (req 21901312->21905407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760748.235473:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484250496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45404us (45878us total) trans 4294972665 rc 0/0 00000100:00100000:1.0:1789760748.235482:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5376 00000100:00100000:1.0:1789760748.235492:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737474010048 time 737520982709 state 0x2 ref 0 00000100:00100000:1.0:1789760748.238381:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760748.238502:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484250624 00000100:00100000:2.0:1789760748.238512:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484250624 00000100:00100000:2.0:1789760748.238569:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737474010048 time 737520982709 state 0x0 ref 1 00000100:00100000:2.0:1789760748.238584:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737521243125 time 737520982709 state 0x1 ref 1 00000100:00100000:2.0:1789760748.238613:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737521243125 time 737569544172 state 0x0 ref 1 00000100:00100000:2.0:1789760748.238624:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484250624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760748.238670:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21905408, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:2.0:1789760748.238679:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.238694:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:2.0:1789760748.238698:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760748.238704:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21905408->21909503] (req 21905408->21909503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.238717:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21905408->21909503] (req 21905408->21909503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760748.275514:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972666 is committed 00002000:00010000:3.0:1789760748.275608:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21905408 -> 21909504 00010000:00010000:3.0:1789760748.275623:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21905408->21909503] (req 21905408->21909503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.275635:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21905408->21909503] (req 21905408->21909503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.275644:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21905408->21909503] (req 21905408->21909503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.275652:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760748.275657:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21905408->21909503] (req 21905408->21909503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.275667:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21905408->21909503] (req 21905408->21909503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.275687:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21905408->21909503] (req 21905408->21909503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.275694:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208bf9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21905408->21909503] (req 21905408->21909503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760748.275785:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484250624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37157us (37409us total) trans 4294972666 rc 0/0 00000100:00100000:3.0:1789760748.275795:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5377 00000100:00100000:3.0:1789760748.275805:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737521243125 time 737569544172 state 0x2 ref 0 00000100:00100000:2.0:1789760748.279524:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760748.279608:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484250752 00000100:00100000:3.0:1789760748.279618:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484250752 00000100:00100000:3.0:1789760748.279671:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737521243125 time 737569544172 state 0x0 ref 1 00000100:00100000:3.0:1789760748.279687:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737569804588 time 737569544172 state 0x1 ref 1 00000100:00100000:3.0:1789760748.279719:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737569804588 time 737610649979 state 0x0 ref 1 00000100:00100000:3.0:1789760748.279732:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484250752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760748.279779:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21909504, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:3.0:1789760748.279787:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.279803:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:3.0:1789760748.279807:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760748.279814:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21909504->21913599] (req 21909504->21913599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.279827:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21909504->21913599] (req 21909504->21913599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760748.313211:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972667 is committed 00002000:00010000:2.0:1789760748.313516:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21909504 -> 21913600 00010000:00010000:2.0:1789760748.313531:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21909504->21913599] (req 21909504->21913599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.313543:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21909504->21913599] (req 21909504->21913599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.313552:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21909504->21913599] (req 21909504->21913599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.313559:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760748.313564:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21909504->21913599] (req 21909504->21913599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.313578:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21909504->21913599] (req 21909504->21913599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.313596:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21909504->21913599] (req 21909504->21913599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.313603:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208c00 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21909504->21913599] (req 21909504->21913599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760748.313822:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484250752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34087us (34303us total) trans 4294972667 rc 0/0 00000100:00100000:2.0:1789760748.313834:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5378 00000100:00100000:2.0:1789760748.313844:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737569804588 time 737610649979 state 0x2 ref 0 00000100:00100000:1.0:1789760748.320831:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760748.320907:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484250880 00000100:00100000:3.0:1789760748.320916:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484250880 00000100:00100000:3.0:1789760748.320964:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737569804588 time 737610649979 state 0x0 ref 1 00000100:00100000:3.0:1789760748.320977:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737610910395 time 737610649979 state 0x1 ref 1 00000100:00100000:3.0:1789760748.321002:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737610910395 time 737651932896 state 0x0 ref 1 00000100:00100000:3.0:1789760748.321013:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484250880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760748.321062:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21913600, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:3.0:1789760748.321158:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.321172:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:3.0:1789760748.321177:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760748.321183:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21913600->21917695] (req 21913600->21917695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.321196:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21913600->21917695] (req 21913600->21917695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760748.362200:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21913600 -> 21917696 00010000:00010000:1.0:1789760748.362219:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21913600->21917695] (req 21913600->21917695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.362232:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21913600->21917695] (req 21913600->21917695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.362240:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21913600->21917695] (req 21913600->21917695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.362247:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760748.362252:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21913600->21917695] (req 21913600->21917695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.362262:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21913600->21917695] (req 21913600->21917695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.362290:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21913600->21917695] (req 21913600->21917695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.362299:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208c07 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21913600->21917695] (req 21913600->21917695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760748.362392:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484250880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41376us (41565us total) trans 4294972668 rc 0/0 00000100:00100000:1.0:1789760748.362403:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5379 00000100:00100000:1.0:1789760748.362413:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737610910395 time 737651932896 state 0x2 ref 0 00000001:00080000:0.0:1789760748.363026:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972668 is committed 00000100:00100000:1.0:1789760748.366838:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760748.366976:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484251008 00000100:00100000:0.0:1789760748.366986:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484251008 00000100:00100000:0.0:1789760748.367049:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737610910395 time 737651932896 state 0x0 ref 1 00000100:00100000:0.0:1789760748.367249:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737652193312 time 737651932896 state 0x1 ref 1 00000100:00100000:0.0:1789760748.367320:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737652193312 time 737698249899 state 0x0 ref 1 00000100:00100000:0.0:1789760748.367334:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484251008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760748.367385:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21917696, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:0.0:1789760748.367394:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.367411:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:0.0:1789760748.367416:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760748.367422:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21917696->21921791] (req 21917696->21921791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.367438:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21917696->21921791] (req 21917696->21921791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760748.401188:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972669 is committed 00002000:00010000:0.0:1789760748.401321:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21917696 -> 21921792 00010000:00010000:0.0:1789760748.401336:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21917696->21921791] (req 21917696->21921791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.401348:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21917696->21921791] (req 21917696->21921791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.401356:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21917696->21921791] (req 21917696->21921791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.401363:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760748.401368:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21917696->21921791] (req 21917696->21921791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.401377:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21917696->21921791] (req 21917696->21921791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.401396:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21917696->21921791] (req 21917696->21921791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.401405:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c0e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21917696->21921791] (req 21917696->21921791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760748.401573:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484251008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34236us (34742us total) trans 4294972669 rc 0/0 00000100:00100000:0.0:1789760748.401583:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5380 00000100:00100000:0.0:1789760748.401592:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737652193312 time 737698249899 state 0x2 ref 0 00000100:00100000:2.0:1789760748.409926:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760748.410010:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484251136 00000100:00100000:3.0:1789760748.410020:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484251136 00000100:00100000:3.0:1789760748.410126:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737652193312 time 737698249899 state 0x0 ref 1 00000100:00100000:3.0:1789760748.410144:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737698510315 time 737698249899 state 0x1 ref 1 00000100:00100000:3.0:1789760748.410167:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737698510315 time 737741097933 state 0x0 ref 1 00000100:00100000:3.0:1789760748.410181:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484251136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760748.410230:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21921792, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:3.0:1789760748.410239:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.410253:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:3.0:1789760748.410258:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760748.410273:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21921792->21925887] (req 21921792->21925887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.410289:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21921792->21925887] (req 21921792->21925887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.432343:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760748.432436:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:2.0:1789760748.432503:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508924416:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1789760748.432569:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760748.432635:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508924416 00000100:00100000:2.0:1789760748.432643:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508924416 00000100:00100000:2.0:1789760748.432693:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 386 00000100:00100000:2.0:1789760748.432702:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508924416:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:2.0:1789760748.432723:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528023552 pending 1703936 free 3836739584 avail 3817865216 00000100:00100000:2.0:1789760748.432766:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508924416:12345-0@lo:13:osp-pre-0-0.0 Request processed in 65us (197us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760748.432776:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 386 00000100:00080000:2.0:1789760748.432795:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760748.432828:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508924416:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1789760748.438057:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508924672:0@lo:13:osp-pre-0-1.0 00000100:00100000:2.0:1789760748.438160:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760748.438255:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:2.0:1789760748.438275:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508924544:0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760748.438284:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508924672 00000100:00100000:3.0:1789760748.438293:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508924672 00000100:00100000:3.0:1789760748.438359:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 387 00000100:00100000:3.0:1789760748.438369:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508924672:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:3.0:1789760748.438392:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528023552 pending 1703936 free 3836739584 avail 3817865216 00000100:00100000:3.0:1789760748.438446:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508924672:12345-0@lo:13:osp-pre-0-1.0 Request processed in 79us (287us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760748.438461:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 387 00000100:00100000:2.0:1789760748.438502:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1789760748.438567:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760748.438625:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508924672:0@lo:13:osp-pre-0-1.0 00000100:00100000:3.0:1789760748.438646:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508924544 00000100:00100000:3.0:1789760748.438651:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508924544 00000100:00100000:3.0:1789760748.438689:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 445 00000100:00100000:3.0:1789760748.438698:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508924544:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:3.0:1789760748.438740:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:3.0:1789760748.439883:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508924544:12345-0@lo:41:osp-pre-0-1.0 Request processed in 1181us (1376us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760748.439893:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 445 00000100:00100000:3.0:1789760748.439939:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98d9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508924544:0@lo:41:osp-pre-0-1.0 00002000:00010000:2.0:1789760748.479763:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21921792 -> 21925888 00010000:00010000:2.0:1789760748.479783:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21921792->21925887] (req 21921792->21925887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.479797:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21921792->21925887] (req 21921792->21925887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.479805:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21921792->21925887] (req 21921792->21925887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.479812:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760748.479817:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21921792->21925887] (req 21921792->21925887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.479825:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21921792->21925887] (req 21921792->21925887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.479850:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21921792->21925887] (req 21921792->21925887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.479859:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208c15 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21921792->21925887] (req 21921792->21925887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760748.479949:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484251136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69767us (70028us total) trans 4294972670 rc 0/0 00000100:00100000:2.0:1789760748.479958:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5381 00000100:00100000:2.0:1789760748.479967:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737698510315 time 737741097933 state 0x2 ref 0 00000001:00080000:1.0:1789760748.480671:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972670 is committed 00000100:00100000:0.0:1789760748.489034:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760748.489212:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484251264 00000100:00100000:2.0:1789760748.489222:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484251264 00000100:00100000:2.0:1789760748.489283:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737698510315 time 737741097933 state 0x0 ref 1 00000100:00100000:2.0:1789760748.489298:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737741358349 time 737741097933 state 0x1 ref 1 00000100:00100000:2.0:1789760748.489395:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737741358349 time 737820324710 state 0x0 ref 1 00000100:00100000:2.0:1789760748.489408:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484251264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760748.489457:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21925888, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:2.0:1789760748.489467:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.489483:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:2.0:1789760748.489489:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760748.489495:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21925888->21929983] (req 21925888->21929983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.489511:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21925888->21929983] (req 21925888->21929983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760748.533173:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972671 is committed 00002000:00010000:3.0:1789760748.533288:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21925888 -> 21929984 00010000:00010000:3.0:1789760748.533304:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21925888->21929983] (req 21925888->21929983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.533317:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21925888->21929983] (req 21925888->21929983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.533324:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21925888->21929983] (req 21925888->21929983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.533331:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760748.533335:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21925888->21929983] (req 21925888->21929983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.533344:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21925888->21929983] (req 21925888->21929983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.533429:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21925888->21929983] (req 21925888->21929983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.533485:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208c1c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21925888->21929983] (req 21925888->21929983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760748.533963:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484251264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44552us (44934us total) trans 4294972671 rc 0/0 00000100:00100000:3.0:1789760748.533975:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5382 00000100:00100000:3.0:1789760748.533985:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737741358349 time 737820324710 state 0x2 ref 0 00000100:00100000:3.0:1789760748.537395:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760748.537534:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484251392 00000100:00100000:1.0:1789760748.537544:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484251392 00000100:00100000:1.0:1789760748.537595:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737741358349 time 737820324710 state 0x0 ref 1 00000100:00100000:1.0:1789760748.537611:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737820585126 time 737820324710 state 0x1 ref 1 00000100:00100000:1.0:1789760748.537644:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737820585126 time 737868574275 state 0x0 ref 1 00000100:00100000:1.0:1789760748.537656:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706c850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484251392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760748.537769:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21929984, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:1.0:1789760748.537779:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.537795:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:1.0:1789760748.537799:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760748.537805:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21929984->21934079] (req 21929984->21934079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.537819:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21929984->21934079] (req 21929984->21934079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760748.584228:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972672 is committed 00002000:00010000:0.0:1789760748.584466:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21929984 -> 21934080 00010000:00010000:0.0:1789760748.584487:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21929984->21934079] (req 21929984->21934079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.584511:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21929984->21934079] (req 21929984->21934079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.584521:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21929984->21934079] (req 21929984->21934079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.584528:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760748.584533:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21929984->21934079] (req 21929984->21934079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.584542:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21929984->21934079] (req 21929984->21934079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.584562:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21929984->21934079] (req 21929984->21934079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.584569:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208c23 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21929984->21934079] (req 21929984->21934079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760748.584784:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706c850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484251392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47124us (47394us total) trans 4294972672 rc 0/0 00000100:00100000:0.0:1789760748.584794:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5383 00000100:00100000:0.0:1789760748.584804:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737820585126 time 737868574275 state 0x2 ref 0 00000100:00100000:3.0:1789760748.594176:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760748.594330:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484251520 00000100:00100000:0.0:1789760748.594340:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484251520 00000100:00100000:0.0:1789760748.594479:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737820585126 time 737868574275 state 0x0 ref 1 00000100:00100000:0.0:1789760748.594494:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737868834691 time 737868574275 state 0x1 ref 1 00000100:00100000:0.0:1789760748.594523:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737868834691 time 737925453990 state 0x0 ref 1 00000100:00100000:0.0:1789760748.594535:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484251520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760748.594583:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21934080, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:0.0:1789760748.594592:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.594607:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:0.0:1789760748.594612:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760748.594617:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21934080->21938175] (req 21934080->21938175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.594631:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21934080->21938175] (req 21934080->21938175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760748.633819:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972673 is committed 00002000:00010000:0.0:1789760748.633917:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21934080 -> 21938176 00010000:00010000:0.0:1789760748.633932:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21934080->21938175] (req 21934080->21938175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.633944:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21934080->21938175] (req 21934080->21938175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.633953:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21934080->21938175] (req 21934080->21938175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.633960:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760748.633965:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21934080->21938175] (req 21934080->21938175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.633972:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21934080->21938175] (req 21934080->21938175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.633991:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21934080->21938175] (req 21934080->21938175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.633998:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c2a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21934080->21938175] (req 21934080->21938175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760748.635059:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484251520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40510us (40878us total) trans 4294972673 rc 0/0 00000100:00100000:0.0:1789760748.635098:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5384 00000100:00100000:0.0:1789760748.635106:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737868834691 time 737925453990 state 0x2 ref 0 00000100:00100000:2.0:1789760748.637581:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760748.637954:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484251648 00000100:00100000:3.0:1789760748.637964:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484251648 00000100:00100000:3.0:1789760748.638019:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737868834691 time 737925453990 state 0x0 ref 1 00000100:00100000:3.0:1789760748.638035:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737925714406 time 737925453990 state 0x1 ref 1 00000100:00100000:3.0:1789760748.638063:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737925714406 time 737968993682 state 0x0 ref 1 00000100:00100000:3.0:1789760748.638110:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484251648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760748.638157:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21938176, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:3.0:1789760748.638166:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.638187:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=737s v=(0 0) 00010000:00010000:3.0:1789760748.638191:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760748.638198:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21938176->21942271] (req 21938176->21942271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.638210:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21938176->21942271] (req 21938176->21942271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760748.673832:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972674 is committed 00002000:00010000:2.0:1789760748.673837:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21938176 -> 21942272 00010000:00010000:2.0:1789760748.673852:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21938176->21942271] (req 21938176->21942271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.673865:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21938176->21942271] (req 21938176->21942271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.673874:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21938176->21942271] (req 21938176->21942271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.673881:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760748.673886:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21938176->21942271] (req 21938176->21942271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.673896:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21938176->21942271] (req 21938176->21942271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.673918:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21938176->21942271] (req 21938176->21942271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.673930:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208c31 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21938176->21942271] (req 21938176->21942271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760748.674023:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484251648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35913us (36447us total) trans 4294972674 rc 0/0 00000100:00100000:2.0:1789760748.674034:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5385 00000100:00100000:2.0:1789760748.674119:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737925714406 time 737968993682 state 0x2 ref 0 00000100:00100000:3.0:1789760748.677444:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760748.678627:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484251776 00000100:00100000:2.0:1789760748.678639:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484251776 00000100:00100000:3.0:1789760748.678761:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737925714406 time 737968993682 state 0x0 ref 1 00000100:00100000:3.0:1789760748.678777:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737969254098 time 737968993682 state 0x1 ref 1 00000100:00100000:3.0:1789760748.678842:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737969254098 time 738009772275 state 0x0 ref 1 00000100:00100000:3.0:1789760748.678855:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484251776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760748.678906:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21942272, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:3.0:1789760748.678913:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.678925:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:3.0:1789760748.678928:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760748.678932:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21942272->21946367] (req 21942272->21946367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.678946:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21942272->21946367] (req 21942272->21946367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760748.711565:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972675 is committed 00002000:00010000:3.0:1789760748.711693:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21942272 -> 21946368 00010000:00010000:3.0:1789760748.711708:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21942272->21946367] (req 21942272->21946367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.711720:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21942272->21946367] (req 21942272->21946367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.711729:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21942272->21946367] (req 21942272->21946367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.711737:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760748.711742:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21942272->21946367] (req 21942272->21946367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.711751:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21942272->21946367] (req 21942272->21946367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.711771:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21942272->21946367] (req 21942272->21946367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.711780:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208c38 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21942272->21946367] (req 21942272->21946367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760748.711876:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484251776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33019us (34437us total) trans 4294972675 rc 0/0 00000100:00100000:3.0:1789760748.711886:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5386 00000100:00100000:3.0:1789760748.711896:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737969254098 time 738009772275 state 0x2 ref 0 00000100:00100000:3.0:1789760748.714509:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760748.714772:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484251904 00000100:00100000:0.0:1789760748.714781:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484251904 00000100:00100000:0.0:1789760748.714836:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 737969254098 time 738009772275 state 0x0 ref 1 00000100:00100000:0.0:1789760748.714851:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738010032691 time 738009772275 state 0x1 ref 1 00000100:00100000:0.0:1789760748.714926:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738010032691 time 738045856140 state 0x0 ref 1 00000100:00100000:0.0:1789760748.714940:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484251904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760748.714988:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21946368, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:0.0:1789760748.714997:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.715012:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:0.0:1789760748.715016:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760748.715023:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21946368->21950463] (req 21946368->21950463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.715036:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21946368->21950463] (req 21946368->21950463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760748.751299:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21946368 -> 21950464 00010000:00010000:0.0:1789760748.751318:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21946368->21950463] (req 21946368->21950463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.751332:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21946368->21950463] (req 21946368->21950463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.751342:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21946368->21950463] (req 21946368->21950463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.751350:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760748.751355:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21946368->21950463] (req 21946368->21950463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.751365:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21946368->21950463] (req 21946368->21950463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.751385:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21946368->21950463] (req 21946368->21950463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.751393:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c3f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21946368->21950463] (req 21946368->21950463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760748.751492:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484251904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36550us (36988us total) trans 4294972676 rc 0/0 00000100:00100000:0.0:1789760748.751506:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5387 00000100:00100000:0.0:1789760748.751515:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738010032691 time 738045856140 state 0x2 ref 0 00000001:00080000:3.0:1789760748.751957:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972676 is committed 00000100:00100000:2.0:1789760748.756200:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760748.757006:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484252032 00000100:00100000:0.0:1789760748.757016:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484252032 00000100:00100000:0.0:1789760748.757148:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738010032691 time 738045856140 state 0x0 ref 1 00000100:00100000:0.0:1789760748.757163:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738046116556 time 738045856140 state 0x1 ref 1 00000100:00100000:0.0:1789760748.757236:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738046116556 time 738088166137 state 0x0 ref 1 00000100:00100000:0.0:1789760748.757251:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484252032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760748.757309:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21950464, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:0.0:1789760748.757319:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.757334:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:0.0:1789760748.757338:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760748.757344:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21950464->21954559] (req 21950464->21954559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.757358:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21950464->21954559] (req 21950464->21954559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760748.783549:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21950464 -> 21954560 00010000:00010000:0.0:1789760748.783569:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21950464->21954559] (req 21950464->21954559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760748.783575:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972677 is committed 00010000:00010000:0.0:1789760748.783582:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21950464->21954559] (req 21950464->21954559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.783592:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21950464->21954559] (req 21950464->21954559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.783599:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760748.783604:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21950464->21954559] (req 21950464->21954559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.783613:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21950464->21954559] (req 21950464->21954559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.783633:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21950464->21954559] (req 21950464->21954559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.783640:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c46 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21950464->21954559] (req 21950464->21954559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760748.783726:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484252032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26474us (27663us total) trans 4294972677 rc 0/0 00000100:00100000:0.0:1789760748.783737:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5388 00000100:00100000:0.0:1789760748.783747:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738046116556 time 738088166137 state 0x2 ref 0 00000100:00100000:2.0:1789760748.786390:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760748.786508:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484252160 00000100:00100000:3.0:1789760748.786518:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484252160 00000100:00100000:3.0:1789760748.786573:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738046116556 time 738088166137 state 0x0 ref 1 00000100:00100000:3.0:1789760748.786590:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738088426553 time 738088166137 state 0x1 ref 1 00000100:00100000:3.0:1789760748.786660:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738088426553 time 738117589775 state 0x0 ref 1 00000100:00100000:3.0:1789760748.786674:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484252160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760748.786723:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21954560, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:3.0:1789760748.786732:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.786749:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:3.0:1789760748.786753:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760748.786759:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21954560->21958655] (req 21954560->21958655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.786779:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21954560->21958655] (req 21954560->21958655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760748.824907:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972678 is committed 00002000:00010000:2.0:1789760748.825785:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21954560 -> 21958656 00010000:00010000:2.0:1789760748.825803:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21954560->21958655] (req 21954560->21958655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.825951:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21954560->21958655] (req 21954560->21958655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.825962:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21954560->21958655] (req 21954560->21958655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.825970:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760748.825975:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21954560->21958655] (req 21954560->21958655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.825987:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21954560->21958655] (req 21954560->21958655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.826008:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21954560->21958655] (req 21954560->21958655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.826016:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208c4d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21954560->21958655] (req 21954560->21958655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760748.826201:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484252160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39525us (39815us total) trans 4294972678 rc 0/0 00000100:00100000:2.0:1789760748.826213:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5389 00000100:00100000:2.0:1789760748.826499:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738088426553 time 738117589775 state 0x2 ref 0 00000100:00100000:3.0:1789760748.830688:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760748.830755:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484252288 00000100:00100000:0.0:1789760748.830766:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484252288 00000100:00100000:0.0:1789760748.830827:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738088426553 time 738117589775 state 0x0 ref 1 00000100:00100000:0.0:1789760748.830843:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738117850191 time 738117589775 state 0x1 ref 1 00000100:00100000:0.0:1789760748.830879:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738117850191 time 738161809275 state 0x0 ref 1 00000100:00100000:0.0:1789760748.830895:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484252288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760748.830945:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21958656, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:0.0:1789760748.830955:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.830971:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:0.0:1789760748.830976:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760748.830982:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21958656->21962751] (req 21958656->21962751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.830997:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21958656->21962751] (req 21958656->21962751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760748.860662:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21958656 -> 21962752 00010000:00010000:3.0:1789760748.860681:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21958656->21962751] (req 21958656->21962751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.860694:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21958656->21962751] (req 21958656->21962751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.860702:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21958656->21962751] (req 21958656->21962751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.860709:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760748.860714:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21958656->21962751] (req 21958656->21962751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.860722:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21958656->21962751] (req 21958656->21962751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.860746:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21958656->21962751] (req 21958656->21962751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760748.860755:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208c54 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21958656->21962751] (req 21958656->21962751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760748.860761:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972679 is committed 00000100:00100000:3.0:1789760748.860863:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484252288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29968us (30180us total) trans 4294972679 rc 0/0 00000100:00100000:3.0:1789760748.860875:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5390 00000100:00100000:3.0:1789760748.860885:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738117850191 time 738161809275 state 0x2 ref 0 00000100:00100000:3.0:1789760748.865143:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760748.865215:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484252416 00000100:00100000:1.0:1789760748.865225:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484252416 00000100:00100000:1.0:1789760748.865288:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738117850191 time 738161809275 state 0x0 ref 1 00000100:00100000:1.0:1789760748.865303:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738162069691 time 738161809275 state 0x1 ref 1 00000100:00100000:1.0:1789760748.865337:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738162069691 time 738196267214 state 0x0 ref 1 00000100:00100000:1.0:1789760748.865351:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484252416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760748.865395:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21962752, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:1.0:1789760748.865404:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.865420:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:1.0:1789760748.865424:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760748.865430:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21962752->21966847] (req 21962752->21966847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.865445:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21962752->21966847] (req 21962752->21966847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760748.894755:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972680 is committed 00002000:00010000:1.0:1789760748.894858:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21962752 -> 21966848 00010000:00010000:1.0:1789760748.894873:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21962752->21966847] (req 21962752->21966847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.894886:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21962752->21966847] (req 21962752->21966847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.894895:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21962752->21966847] (req 21962752->21966847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.894901:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760748.894906:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21962752->21966847] (req 21962752->21966847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.894915:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21962752->21966847] (req 21962752->21966847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.894935:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21962752->21966847] (req 21962752->21966847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.894942:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c5b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21962752->21966847] (req 21962752->21966847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760748.895034:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484252416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29681us (29894us total) trans 4294972680 rc 0/0 00000100:00100000:1.0:1789760748.895157:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5391 00000100:00100000:1.0:1789760748.895167:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738162069691 time 738196267214 state 0x2 ref 0 00000100:00100000:2.0:1789760748.897980:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760748.898150:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484252544 00000100:00100000:1.0:1789760748.898159:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484252544 00000100:00100000:1.0:1789760748.898207:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738162069691 time 738196267214 state 0x0 ref 1 00000100:00100000:1.0:1789760748.898222:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738196527630 time 738196267214 state 0x1 ref 1 00000100:00100000:1.0:1789760748.898250:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738196527630 time 738229181016 state 0x0 ref 1 00000100:00100000:1.0:1789760748.898273:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484252544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760748.898322:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21966848, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:1.0:1789760748.898331:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.898346:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:1.0:1789760748.898357:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760748.898364:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21966848->21970943] (req 21966848->21970943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.898379:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21966848->21970943] (req 21966848->21970943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760748.921205:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972681 is committed 00002000:00010000:1.0:1789760748.921254:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21966848 -> 21970944 00010000:00010000:1.0:1789760748.921282:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21966848->21970943] (req 21966848->21970943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.921412:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21966848->21970943] (req 21966848->21970943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.921423:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21966848->21970943] (req 21966848->21970943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.921434:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760748.921439:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21966848->21970943] (req 21966848->21970943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.921449:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21966848->21970943] (req 21966848->21970943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.921469:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21966848->21970943] (req 21966848->21970943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.921477:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208c62 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21966848->21970943] (req 21966848->21970943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760748.921561:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484252544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23296us (23586us total) trans 4294972681 rc 0/0 00000100:00100000:1.0:1789760748.921572:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5392 00000100:00100000:1.0:1789760748.921582:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738196527630 time 738229181016 state 0x2 ref 0 00000100:00100000:3.0:1789760748.926716:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760748.926877:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484252672 00000100:00100000:2.0:1789760748.926887:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484252672 00000100:00100000:2.0:1789760748.926941:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738196527630 time 738229181016 state 0x0 ref 1 00000100:00100000:2.0:1789760748.926956:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738229441432 time 738229181016 state 0x1 ref 1 00000100:00100000:2.0:1789760748.926987:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738229441432 time 738257918062 state 0x0 ref 1 00000100:00100000:2.0:1789760748.926999:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484252672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760748.927055:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21970944, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:2.0:1789760748.927094:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.927112:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:2.0:1789760748.927116:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760748.927123:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21970944->21975039] (req 21970944->21975039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760748.927137:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21970944->21975039] (req 21970944->21975039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760748.941408:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4f12a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508924800:0@lo:41:osp-pre-1-0.0 00000100:00100000:3.0:1789760748.941479:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760748.941787:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508924800 00000100:00100000:0.0:1789760748.941798:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508924800 00000100:00100000:0.0:1789760748.941997:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 446 00000100:00100000:0.0:1789760748.942013:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508924800:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:0.0:1789760748.942506:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:0.0:1789760748.942581:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508924800:12345-0@lo:41:osp-pre-1-0.0 Request processed in 572us (1100us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760748.942593:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 446 00000100:00100000:1.0:1789760748.942841:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4f12a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508924800:0@lo:41:osp-pre-1-0.0 00000001:00080000:3.0:1789760748.957058:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972682 is committed 00002000:00010000:1.0:1789760748.957232:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21970944 -> 21975040 00010000:00010000:1.0:1789760748.957247:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21970944->21975039] (req 21970944->21975039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.957267:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21970944->21975039] (req 21970944->21975039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.957278:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21970944->21975039] (req 21970944->21975039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.957285:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760748.957290:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21970944->21975039] (req 21970944->21975039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.957300:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21970944->21975039] (req 21970944->21975039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.957330:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21970944->21975039] (req 21970944->21975039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760748.957339:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208c69 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21970944->21975039] (req 21970944->21975039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760748.957436:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484252672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30434us (30726us total) trans 4294972682 rc 0/0 00000100:00100000:1.0:1789760748.957446:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5393 00000100:00100000:1.0:1789760748.957455:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738229441432 time 738257918062 state 0x2 ref 0 00000100:00100000:3.0:1789760748.960871:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760748.961164:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484252800 00000100:00100000:0.0:1789760748.961174:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484252800 00000100:00100000:0.0:1789760748.961233:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738229441432 time 738257918062 state 0x0 ref 1 00000100:00100000:0.0:1789760748.961248:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738258178478 time 738257918062 state 0x1 ref 1 00000100:00100000:0.0:1789760748.961315:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738258178478 time 738292245134 state 0x0 ref 1 00000100:00100000:0.0:1789760748.961329:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484252800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760748.961386:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21975040, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:0.0:1789760748.961396:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.961411:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:0.0:1789760748.961415:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760748.961421:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21975040->21979135] (req 21975040->21979135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.961434:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21975040->21979135] (req 21975040->21979135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760748.998400:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21975040 -> 21979136 00000001:00080000:1.0:1789760748.998427:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972683 is committed 00010000:00010000:0.0:1789760748.998429:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21975040->21979135] (req 21975040->21979135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.998457:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21975040->21979135] (req 21975040->21979135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.998467:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21975040->21979135] (req 21975040->21979135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.998475:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760748.998480:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21975040->21979135] (req 21975040->21979135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.998490:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21975040->21979135] (req 21975040->21979135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.998510:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21975040->21979135] (req 21975040->21979135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760748.998519:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c70 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21975040->21979135] (req 21975040->21979135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760748.998617:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484252800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37288us (37751us total) trans 4294972683 rc 0/0 00000100:00100000:0.0:1789760748.998627:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5394 00000100:00100000:0.0:1789760748.998637:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738258178478 time 738292245134 state 0x2 ref 0 00000100:00100000:2.0:1789760749.002172:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760749.002321:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484252928 00000100:00100000:1.0:1789760749.002330:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484252928 00000100:00100000:1.0:1789760749.002380:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738258178478 time 738292245134 state 0x0 ref 1 00000100:00100000:1.0:1789760749.002391:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738292505550 time 738292245134 state 0x1 ref 1 00000100:00100000:1.0:1789760749.002419:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738292505550 time 738333351125 state 0x0 ref 1 00000100:00100000:1.0:1789760749.002428:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484252928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760749.002469:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21979136, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:1.0:1789760749.002476:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.002488:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:1.0:1789760749.002491:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760749.002496:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21979136->21983231] (req 21979136->21983231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.002506:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21979136->21983231] (req 21979136->21983231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760749.030489:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21979136 -> 21983232 00010000:00010000:1.0:1789760749.030509:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21979136->21983231] (req 21979136->21983231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.030526:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21979136->21983231] (req 21979136->21983231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.030536:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21979136->21983231] (req 21979136->21983231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.030543:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760749.030548:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21979136->21983231] (req 21979136->21983231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.030557:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21979136->21983231] (req 21979136->21983231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.030576:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21979136->21983231] (req 21979136->21983231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.030583:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208c77 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21979136->21983231] (req 21979136->21983231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760749.030669:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484252928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28236us (28502us total) trans 4294972684 rc 0/0 00000100:00100000:1.0:1789760749.030680:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5395 00000100:00100000:1.0:1789760749.030691:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738292505550 time 738333351125 state 0x2 ref 0 00000001:00080000:1.0:1789760749.030740:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972684 is committed 00000100:00100000:1.0:1789760749.036561:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760749.036727:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484253056 00000100:00100000:0.0:1789760749.036737:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484253056 00000100:00100000:0.0:1789760749.036796:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738292505550 time 738333351125 state 0x0 ref 1 00000100:00100000:0.0:1789760749.036812:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738333611541 time 738333351125 state 0x1 ref 1 00000100:00100000:0.0:1789760749.036844:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738333611541 time 738367774223 state 0x0 ref 1 00000100:00100000:0.0:1789760749.036857:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484253056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760749.036903:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21983232, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:0.0:1789760749.036912:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.036927:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:0.0:1789760749.036932:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760749.036938:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21983232->21987327] (req 21983232->21987327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.036952:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21983232->21987327] (req 21983232->21987327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760749.084783:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21983232 -> 21987328 00010000:00010000:1.0:1789760749.084802:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21983232->21987327] (req 21983232->21987327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.084817:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21983232->21987327] (req 21983232->21987327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.084826:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21983232->21987327] (req 21983232->21987327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.084834:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760749.084839:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21983232->21987327] (req 21983232->21987327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.084848:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21983232->21987327] (req 21983232->21987327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.084867:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21983232->21987327] (req 21983232->21987327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.084875:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208c7e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21983232->21987327] (req 21983232->21987327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760749.084970:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484253056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48111us (48415us total) trans 4294972685 rc 0/0 00000100:00100000:1.0:1789760749.084980:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5396 00000100:00100000:1.0:1789760749.084990:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738333611541 time 738367774223 state 0x2 ref 0 00000001:00080000:1.0:1789760749.085029:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972685 is committed 00000100:00100000:3.0:1789760749.088008:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760749.088095:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484253184 00000100:00100000:2.0:1789760749.088104:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484253184 00000100:00100000:2.0:1789760749.088156:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738333611541 time 738367774223 state 0x0 ref 1 00000100:00100000:2.0:1789760749.088172:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738368034639 time 738367774223 state 0x1 ref 1 00000100:00100000:2.0:1789760749.088198:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738368034639 time 738419129095 state 0x0 ref 1 00000100:00100000:2.0:1789760749.088210:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484253184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.088266:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21987328, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:2.0:1789760749.088276:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.088293:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:2.0:1789760749.088298:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.088305:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21987328->21991423] (req 21987328->21991423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.088319:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21987328->21991423] (req 21987328->21991423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760749.118213:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972686 is committed 00002000:00010000:2.0:1789760749.118438:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21987328 -> 21991424 00010000:00010000:2.0:1789760749.118522:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21987328->21991423] (req 21987328->21991423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.118536:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21987328->21991423] (req 21987328->21991423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.118545:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21987328->21991423] (req 21987328->21991423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.118552:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760749.118557:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21987328->21991423] (req 21987328->21991423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.118566:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21987328->21991423] (req 21987328->21991423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.118585:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21987328->21991423] (req 21987328->21991423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.118592:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c85 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21987328->21991423] (req 21987328->21991423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760749.118683:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484253184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30470us (30679us total) trans 4294972686 rc 0/0 00000100:00100000:2.0:1789760749.118694:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5397 00000100:00100000:2.0:1789760749.118703:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738368034639 time 738419129095 state 0x2 ref 0 00000100:00100000:2.0:1789760749.122851:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760749.122961:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484253312 00000100:00100000:3.0:1789760749.122969:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484253312 00000100:00100000:3.0:1789760749.123015:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738368034639 time 738419129095 state 0x0 ref 1 00000100:00100000:3.0:1789760749.123026:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738419389511 time 738419129095 state 0x1 ref 1 00000100:00100000:3.0:1789760749.123147:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738419389511 time 738454077057 state 0x0 ref 1 00000100:00100000:3.0:1789760749.123161:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484253312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760749.123219:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21991424, mtime: 0x6aad94ec, atime: 0x0, ctime: 0x6aad94ec, blocks: 0xa807 00010000:00010000:3.0:1789760749.123228:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.123245:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:3.0:1789760749.123249:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760749.123256:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21991424->21995519] (req 21991424->21995519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.123281:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21991424->21995519] (req 21991424->21995519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760749.150480:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972687 is committed 00002000:00010000:3.0:1789760749.150574:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21991424 -> 21995520 00002000:00010000:3.0:1789760749.150579:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760748 -> 1789760749 00002000:00010000:3.0:1789760749.150582:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760748 -> 1789760749 00010000:00010000:3.0:1789760749.150594:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21991424->21995519] (req 21991424->21995519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.150606:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21991424->21995519] (req 21991424->21995519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.150614:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21991424->21995519] (req 21991424->21995519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.150620:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760749.150625:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21991424->21995519] (req 21991424->21995519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.150633:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21991424->21995519] (req 21991424->21995519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.150651:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21991424->21995519] (req 21991424->21995519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.150659:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208c8c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21991424->21995519] (req 21991424->21995519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760749.150786:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484253312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27623us (27940us total) trans 4294972687 rc 0/0 00000100:00100000:3.0:1789760749.150796:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5398 00000100:00100000:3.0:1789760749.150807:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738419389511 time 738454077057 state 0x2 ref 0 00000100:00100000:1.0:1789760749.153980:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760749.154117:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484253440 00000100:00100000:2.0:1789760749.154127:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484253440 00000100:00100000:2.0:1789760749.154179:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738419389511 time 738454077057 state 0x0 ref 1 00000100:00100000:2.0:1789760749.154342:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738454337473 time 738454077057 state 0x1 ref 1 00000100:00100000:2.0:1789760749.154408:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738454337473 time 738485337370 state 0x0 ref 1 00000100:00100000:2.0:1789760749.154421:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484253440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.154469:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21995520, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xa807 00010000:00010000:2.0:1789760749.154478:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.154493:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:2.0:1789760749.154497:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.154504:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21995520->21999615] (req 21995520->21999615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.154523:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21995520->21999615] (req 21995520->21999615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760749.204513:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972688 is committed 00002000:00010000:2.0:1789760749.204655:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21995520 -> 21999616 00010000:00010000:2.0:1789760749.204670:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21995520->21999615] (req 21995520->21999615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.204684:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21995520->21999615] (req 21995520->21999615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.204693:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21995520->21999615] (req 21995520->21999615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.204701:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760749.204706:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21995520->21999615] (req 21995520->21999615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.204716:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21995520->21999615] (req 21995520->21999615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.204735:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21995520->21999615] (req 21995520->21999615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.204744:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208c93 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21995520->21999615] (req 21995520->21999615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760749.204879:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484253440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50455us (50903us total) trans 4294972688 rc 0/0 00000100:00100000:2.0:1789760749.204891:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5399 00000100:00100000:2.0:1789760749.204901:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738454337473 time 738485337370 state 0x2 ref 0 00000100:00100000:0.0:1789760749.217324:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760749.217439:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484253568 00000100:00100000:1.0:1789760749.217449:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484253568 00000100:00100000:1.0:1789760749.217519:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738454337473 time 738485337370 state 0x0 ref 1 00000100:00100000:1.0:1789760749.217535:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738485597786 time 738485337370 state 0x1 ref 1 00000100:00100000:1.0:1789760749.217563:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738485597786 time 738548494316 state 0x0 ref 1 00000100:00100000:1.0:1789760749.217575:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484253568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760749.217623:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 21999616, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xa807 00010000:00010000:1.0:1789760749.217632:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.217648:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:1.0:1789760749.217653:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760749.217659:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21999616->22003711] (req 21999616->22003711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.217673:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21999616->22003711] (req 21999616->22003711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760749.267264:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972689 is committed 00002000:00010000:2.0:1789760749.267382:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 21999616 -> 22003712 00010000:00010000:2.0:1789760749.267397:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21999616->22003711] (req 21999616->22003711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.267410:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21999616->22003711] (req 21999616->22003711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.267419:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21999616->22003711] (req 21999616->22003711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.267426:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760749.267431:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21999616->22003711] (req 21999616->22003711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.267440:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21999616->22003711] (req 21999616->22003711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.267459:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21999616->22003711] (req 21999616->22003711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.267470:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208c9a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [21999616->22003711] (req 21999616->22003711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760749.267566:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484253568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49988us (50248us total) trans 4294972689 rc 0/0 00000100:00100000:2.0:1789760749.267577:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5400 00000100:00100000:2.0:1789760749.268163:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738485597786 time 738548494316 state 0x2 ref 0 00000100:00100000:3.0:1789760749.271655:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760749.271752:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484253696 00000100:00100000:1.0:1789760749.271763:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484253696 00000100:00100000:1.0:1789760749.271819:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738485597786 time 738548494316 state 0x0 ref 1 00000100:00100000:1.0:1789760749.271834:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738548754732 time 738548494316 state 0x1 ref 1 00000100:00100000:1.0:1789760749.271864:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738548754732 time 738602794949 state 0x0 ref 1 00000100:00100000:1.0:1789760749.271876:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defde300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484253696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760749.271923:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22003712, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xa807 00010000:00010000:1.0:1789760749.271933:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.271948:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:1.0:1789760749.271953:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760749.271959:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22003712->22007807] (req 22003712->22007807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.271972:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22003712->22007807] (req 22003712->22007807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760749.316726:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972690 is committed 00002000:00010000:1.0:1789760749.316805:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22003712 -> 22007808 00010000:00010000:1.0:1789760749.316820:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22003712->22007807] (req 22003712->22007807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.316832:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22003712->22007807] (req 22003712->22007807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.316847:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22003712->22007807] (req 22003712->22007807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.316854:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760749.316858:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22003712->22007807] (req 22003712->22007807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.316867:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22003712->22007807] (req 22003712->22007807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.316886:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22003712->22007807] (req 22003712->22007807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.316894:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208ca1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22003712->22007807] (req 22003712->22007807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760749.317026:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defde300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484253696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45148us (45377us total) trans 4294972690 rc 0/0 00000100:00100000:1.0:1789760749.317036:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5401 00000100:00100000:1.0:1789760749.317872:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738548754732 time 738602794949 state 0x2 ref 0 00000100:00100000:1.0:1789760749.320736:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760749.320809:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484253824 00000100:00100000:3.0:1789760749.320819:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484253824 00000100:00100000:3.0:1789760749.320869:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738548754732 time 738602794949 state 0x0 ref 1 00000100:00100000:3.0:1789760749.320882:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738603055365 time 738602794949 state 0x1 ref 1 00000100:00100000:3.0:1789760749.320913:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738603055365 time 738651844619 state 0x0 ref 1 00000100:00100000:3.0:1789760749.320926:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484253824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760749.320976:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22007808, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xa807 00010000:00010000:3.0:1789760749.320986:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.321007:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:3.0:1789760749.321011:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760749.321027:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22007808->22011903] (req 22007808->22011903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.321046:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22007808->22011903] (req 22007808->22011903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760749.352995:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22007808 -> 22011904 00010000:00010000:3.0:1789760749.353015:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22007808->22011903] (req 22007808->22011903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.353027:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22007808->22011903] (req 22007808->22011903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.353036:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22007808->22011903] (req 22007808->22011903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.353138:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760749.353143:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22007808->22011903] (req 22007808->22011903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.353152:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22007808->22011903] (req 22007808->22011903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.353172:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22007808->22011903] (req 22007808->22011903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.353246:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208ca8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22007808->22011903] (req 22007808->22011903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760749.353429:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484253824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32419us (32615us total) trans 4294972691 rc 0/0 00000100:00100000:3.0:1789760749.353443:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5402 00000100:00100000:3.0:1789760749.353454:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738603055365 time 738651844619 state 0x2 ref 0 00000001:00080000:2.0:1789760749.356473:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972691 is committed 00000100:00100000:0.0:1789760749.357544:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760749.357639:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484253952 00000100:00100000:2.0:1789760749.357648:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484253952 00000100:00100000:2.0:1789760749.357697:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738603055365 time 738651844619 state 0x0 ref 1 00000100:00100000:2.0:1789760749.357712:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738652105035 time 738651844619 state 0x1 ref 1 00000100:00100000:2.0:1789760749.357781:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738652105035 time 738688711113 state 0x0 ref 1 00000100:00100000:2.0:1789760749.357793:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484253952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.357849:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22011904, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xa807 00010000:00010000:2.0:1789760749.357859:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.357875:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:2.0:1789760749.357879:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.357887:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22011904->22015999] (req 22011904->22015999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.357900:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22011904->22015999] (req 22011904->22015999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760749.389004:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972692 is committed 00002000:00010000:2.0:1789760749.389181:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22011904 -> 22016000 00010000:00010000:2.0:1789760749.389198:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22011904->22015999] (req 22011904->22015999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.389211:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22011904->22015999] (req 22011904->22015999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.389221:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22011904->22015999] (req 22011904->22015999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.389229:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760749.389234:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22011904->22015999] (req 22011904->22015999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.389243:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22011904->22015999] (req 22011904->22015999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.389272:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22011904->22015999] (req 22011904->22015999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.389280:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208caf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22011904->22015999] (req 22011904->22015999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760749.389476:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484253952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31680us (31934us total) trans 4294972692 rc 0/0 00000100:00100000:2.0:1789760749.389487:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5403 00000100:00100000:2.0:1789760749.389497:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738652105035 time 738688711113 state 0x2 ref 0 00000100:00100000:1.0:1789760749.392939:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760749.393011:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484254080 00000100:00100000:2.0:1789760749.393020:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484254080 00000100:00100000:2.0:1789760749.393115:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738652105035 time 738688711113 state 0x0 ref 1 00000100:00100000:2.0:1789760749.393131:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738688971529 time 738688711113 state 0x1 ref 1 00000100:00100000:2.0:1789760749.393159:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738688971529 time 738724090030 state 0x0 ref 1 00000100:00100000:2.0:1789760749.393172:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484254080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.393221:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22016000, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xa807 00010000:00010000:2.0:1789760749.393230:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.393246:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:2.0:1789760749.393251:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.393271:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22016000->22020095] (req 22016000->22020095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.393285:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22016000->22020095] (req 22016000->22020095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760749.421813:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972693 is committed 00002000:00010000:2.0:1789760749.421964:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22016000 -> 22020096 00010000:00010000:2.0:1789760749.421981:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22016000->22020095] (req 22016000->22020095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.421994:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22016000->22020095] (req 22016000->22020095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.422004:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22016000->22020095] (req 22016000->22020095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.422010:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760749.422015:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22016000->22020095] (req 22016000->22020095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.422023:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22016000->22020095] (req 22016000->22020095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.422411:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22016000->22020095] (req 22016000->22020095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.422421:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cb6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22016000->22020095] (req 22016000->22020095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760749.422549:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484254080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29375us (29615us total) trans 4294972693 rc 0/0 00000100:00100000:2.0:1789760749.422563:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5404 00000100:00100000:2.0:1789760749.422573:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738688971529 time 738724090030 state 0x2 ref 0 00000100:00100000:1.0:1789760749.426452:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760749.426605:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484254208 00000100:00100000:2.0:1789760749.426616:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484254208 00000100:00100000:2.0:1789760749.426688:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738688971529 time 738724090030 state 0x0 ref 1 00000100:00100000:2.0:1789760749.426705:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738724350446 time 738724090030 state 0x1 ref 1 00000100:00100000:2.0:1789760749.426737:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738724350446 time 738757668446 state 0x0 ref 1 00000100:00100000:2.0:1789760749.426750:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484254208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.426802:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22020096, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xa807 00010000:00010000:2.0:1789760749.426811:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.426826:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:2.0:1789760749.426830:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.426836:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22020096->22024191] (req 22020096->22024191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.426849:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22020096->22024191] (req 22020096->22024191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.452309:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760749.468478:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972694 is committed 00002000:00010000:1.0:1789760749.468483:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22020096 -> 22024192 00002000:00010000:1.0:1789760749.468490:0:8106:0:(ofd_lvb.c:331:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb blocks from disk: 43015 -> 45063 00010000:00010000:1.0:1789760749.468504:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22020096->22024191] (req 22020096->22024191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.468519:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22020096->22024191] (req 22020096->22024191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.468529:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22020096->22024191] (req 22020096->22024191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.468537:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760749.468542:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22020096->22024191] (req 22020096->22024191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.468552:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22020096->22024191] (req 22020096->22024191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.468572:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22020096->22024191] (req 22020096->22024191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.468581:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396208cbd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22020096->22024191] (req 22020096->22024191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760749.468675:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484254208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41923us (42228us total) trans 4294972694 rc 0/0 00000100:00100000:1.0:1789760749.468687:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5405 00000100:00100000:1.0:1789760749.468697:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738724350446 time 738757668446 state 0x2 ref 0 00000100:00100000:0.0:1789760749.473802:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760749.474282:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484254336 00000100:00100000:3.0:1789760749.474293:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484254336 00000100:00100000:3.0:1789760749.474359:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738724350446 time 738757668446 state 0x0 ref 1 00000100:00100000:3.0:1789760749.474375:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738757928862 time 738757668446 state 0x1 ref 1 00000100:00100000:3.0:1789760749.474406:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738757928862 time 738805337203 state 0x0 ref 1 00000100:00100000:3.0:1789760749.474419:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484254336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760749.474471:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22024192, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:3.0:1789760749.474480:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.474495:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:3.0:1789760749.474500:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760749.474506:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22024192->22028287] (req 22024192->22028287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.474522:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22024192->22028287] (req 22024192->22028287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760749.504880:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22024192 -> 22028288 00000001:00080000:3.0:1789760749.504884:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972695 is committed 00010000:00010000:1.0:1789760749.504899:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22024192->22028287] (req 22024192->22028287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.504913:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22024192->22028287] (req 22024192->22028287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.504922:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22024192->22028287] (req 22024192->22028287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.504930:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760749.504935:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22024192->22028287] (req 22024192->22028287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.504944:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22024192->22028287] (req 22024192->22028287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.505106:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22024192->22028287] (req 22024192->22028287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.505115:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208cc4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22024192->22028287] (req 22024192->22028287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760749.505290:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484254336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30869us (31493us total) trans 4294972695 rc 0/0 00000100:00100000:1.0:1789760749.505305:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5406 00000100:00100000:1.0:1789760749.505314:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738757928862 time 738805337203 state 0x2 ref 0 00000100:00100000:3.0:1789760749.508869:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760749.509146:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484254464 00000100:00100000:2.0:1789760749.509156:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484254464 00000100:00100000:2.0:1789760749.509214:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738757928862 time 738805337203 state 0x0 ref 1 00000100:00100000:2.0:1789760749.509229:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738805597619 time 738805337203 state 0x1 ref 1 00000100:00100000:2.0:1789760749.509297:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738805597619 time 738840227330 state 0x0 ref 1 00000100:00100000:2.0:1789760749.509309:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defded80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484254464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.509359:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22028288, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:2.0:1789760749.509369:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.509385:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:2.0:1789760749.509389:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.509395:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22028288->22032383] (req 22028288->22032383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.509408:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22028288->22032383] (req 22028288->22032383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760749.543698:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972696 is committed 00002000:00010000:2.0:1789760749.543798:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22028288 -> 22032384 00010000:00010000:2.0:1789760749.543812:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22028288->22032383] (req 22028288->22032383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.543824:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22028288->22032383] (req 22028288->22032383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.543833:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22028288->22032383] (req 22028288->22032383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.543840:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760749.543845:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22028288->22032383] (req 22028288->22032383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.543854:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22028288->22032383] (req 22028288->22032383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.543874:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22028288->22032383] (req 22028288->22032383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.543881:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ccb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22028288->22032383] (req 22028288->22032383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760749.543977:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defded80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484254464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34664us (35109us total) trans 4294972696 rc 0/0 00000100:00100000:2.0:1789760749.544552:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5407 00000100:00100000:2.0:1789760749.544564:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738805597619 time 738840227330 state 0x2 ref 0 00000100:00100000:3.0:1789760749.549763:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760749.549848:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484254592 00000100:00100000:2.0:1789760749.549858:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484254592 00000100:00100000:2.0:1789760749.549909:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738805597619 time 738840227330 state 0x0 ref 1 00000100:00100000:2.0:1789760749.549924:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738840487746 time 738840227330 state 0x1 ref 1 00000100:00100000:2.0:1789760749.549955:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738840487746 time 738880886085 state 0x0 ref 1 00000100:00100000:2.0:1789760749.549971:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484254592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.550032:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22032384, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:2.0:1789760749.550344:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.550365:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:2.0:1789760749.550370:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.550377:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22032384->22036479] (req 22032384->22036479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.550391:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22032384->22036479] (req 22032384->22036479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760749.586630:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22032384 -> 22036480 00010000:00010000:2.0:1789760749.586650:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22032384->22036479] (req 22032384->22036479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.586663:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22032384->22036479] (req 22032384->22036479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.586672:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22032384->22036479] (req 22032384->22036479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.586679:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760749.586685:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22032384->22036479] (req 22032384->22036479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.586695:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22032384->22036479] (req 22032384->22036479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.586712:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22032384->22036479] (req 22032384->22036479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.586720:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208cd2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22032384->22036479] (req 22032384->22036479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760749.586802:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484254592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36829us (37043us total) trans 4294972697 rc 0/0 00000100:00100000:2.0:1789760749.586813:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5408 00000100:00100000:2.0:1789760749.586823:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738840487746 time 738880886085 state 0x2 ref 0 00000001:00080000:1.0:1789760749.587342:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972697 is committed 00000100:00100000:3.0:1789760749.592039:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760749.593056:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484254720 00000100:00100000:1.0:1789760749.593089:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484254720 00000100:00100000:1.0:1789760749.593145:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738840487746 time 738880886085 state 0x0 ref 1 00000100:00100000:1.0:1789760749.593160:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738881146501 time 738880886085 state 0x1 ref 1 00000100:00100000:1.0:1789760749.593252:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738881146501 time 738924181621 state 0x0 ref 1 00000100:00100000:1.0:1789760749.593277:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484254720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760749.593325:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22036480, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:1.0:1789760749.593333:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.593350:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:1.0:1789760749.593354:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760749.593361:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22036480->22040575] (req 22036480->22040575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.593374:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22036480->22040575] (req 22036480->22040575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760749.651779:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972698 is committed 00002000:00010000:1.0:1789760749.651884:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22036480 -> 22040576 00010000:00010000:1.0:1789760749.651900:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22036480->22040575] (req 22036480->22040575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.651913:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22036480->22040575] (req 22036480->22040575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.651922:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22036480->22040575] (req 22036480->22040575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.651931:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760749.651936:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22036480->22040575] (req 22036480->22040575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.651946:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22036480->22040575] (req 22036480->22040575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.651966:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22036480->22040575] (req 22036480->22040575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.651974:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208cd9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22036480->22040575] (req 22036480->22040575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760749.652146:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484254720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58866us (60110us total) trans 4294972698 rc 0/0 00000100:00100000:1.0:1789760749.652160:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5409 00000100:00100000:1.0:1789760749.652171:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738881146501 time 738924181621 state 0x2 ref 0 00000100:00100000:2.0:1789760749.660278:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760749.660414:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484254848 00000100:00100000:1.0:1789760749.660425:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484254848 00000100:00100000:1.0:1789760749.660495:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738881146501 time 738924181621 state 0x0 ref 1 00000100:00100000:1.0:1789760749.660509:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738924442037 time 738924181621 state 0x1 ref 1 00000100:00100000:1.0:1789760749.660539:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738924442037 time 738991469291 state 0x0 ref 1 00000100:00100000:1.0:1789760749.660551:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484254848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760749.660598:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22040576, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:1.0:1789760749.660606:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.660623:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=738s v=(0 0) 00010000:00010000:1.0:1789760749.660627:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760749.660634:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22040576->22044671] (req 22040576->22044671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.660648:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22040576->22044671] (req 22040576->22044671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760749.717809:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972699 is committed 00002000:00010000:0.0:1789760749.717924:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22040576 -> 22044672 00010000:00010000:0.0:1789760749.717940:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22040576->22044671] (req 22040576->22044671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.717958:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22040576->22044671] (req 22040576->22044671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.717966:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22040576->22044671] (req 22040576->22044671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.717973:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760749.717979:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22040576->22044671] (req 22040576->22044671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.717987:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22040576->22044671] (req 22040576->22044671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.718011:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22040576->22044671] (req 22040576->22044671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.718019:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396208ce0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22040576->22044671] (req 22040576->22044671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760749.718148:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484254848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57594us (57887us total) trans 4294972699 rc 0/0 00000100:00100000:0.0:1789760749.718158:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5410 00000100:00100000:0.0:1789760749.718168:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738924442037 time 738991469291 state 0x2 ref 0 00000100:00100000:2.0:1789760749.727897:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760749.730438:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484254976 00000100:00100000:3.0:1789760749.730520:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484254976 00000100:00100000:2.0:1789760749.733705:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738924442037 time 738991469291 state 0x0 ref 1 00000100:00100000:2.0:1789760749.733723:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738991729707 time 738991469291 state 0x1 ref 1 00000100:00100000:2.0:1789760749.733764:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738991729707 time 739064695849 state 0x0 ref 1 00000100:00100000:2.0:1789760749.733778:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484254976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.733832:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22044672, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:2.0:1789760749.733842:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.733857:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:2.0:1789760749.733861:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.733867:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22044672->22048767] (req 22044672->22048767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.733879:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22044672->22048767] (req 22044672->22048767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760749.772982:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972700 is committed 00002000:00010000:0.0:1789760749.773194:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22044672 -> 22048768 00010000:00010000:0.0:1789760749.773210:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22044672->22048767] (req 22044672->22048767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.773222:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22044672->22048767] (req 22044672->22048767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.773230:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22044672->22048767] (req 22044672->22048767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.773237:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760749.773241:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22044672->22048767] (req 22044672->22048767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.773250:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22044672->22048767] (req 22044672->22048767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.773281:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22044672->22048767] (req 22044672->22048767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.773288:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208ce7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22044672->22048767] (req 22044672->22048767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760749.774422:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484254976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39671us (45558us total) trans 4294972700 rc 0/0 00000100:00100000:0.0:1789760749.774435:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5411 00000100:00100000:0.0:1789760749.774512:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738991729707 time 739064695849 state 0x2 ref 0 00000100:00100000:3.0:1789760749.777232:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760749.777305:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484255104 00000100:00100000:2.0:1789760749.777316:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484255104 00000100:00100000:2.0:1789760749.777380:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 738991729707 time 739064695849 state 0x0 ref 1 00000100:00100000:2.0:1789760749.777396:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739064956265 time 739064695849 state 0x1 ref 1 00000100:00100000:2.0:1789760749.777424:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739064956265 time 739108354937 state 0x0 ref 1 00000100:00100000:2.0:1789760749.777436:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484255104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.777598:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22048768, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:2.0:1789760749.777608:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.777625:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:2.0:1789760749.777630:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.777637:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22048768->22052863] (req 22048768->22052863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.777650:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22048768->22052863] (req 22048768->22052863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760749.806891:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972701 is committed 00002000:00010000:2.0:1789760749.806976:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22048768 -> 22052864 00010000:00010000:2.0:1789760749.806991:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22048768->22052863] (req 22048768->22052863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.807004:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22048768->22052863] (req 22048768->22052863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.807012:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22048768->22052863] (req 22048768->22052863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.807018:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760749.807023:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22048768->22052863] (req 22048768->22052863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.807031:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22048768->22052863] (req 22048768->22052863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.807144:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22048768->22052863] (req 22048768->22052863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.807152:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208cee lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22048768->22052863] (req 22048768->22052863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760749.807298:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484255104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29859us (30071us total) trans 4294972701 rc 0/0 00000100:00100000:2.0:1789760749.807309:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5412 00000100:00100000:2.0:1789760749.807318:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739064956265 time 739108354937 state 0x2 ref 0 00000100:00100000:3.0:1789760749.811271:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760749.811337:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484255232 00000100:00100000:1.0:1789760749.811352:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484255232 00000100:00100000:1.0:1789760749.811401:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739064956265 time 739108354937 state 0x0 ref 1 00000100:00100000:1.0:1789760749.811415:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739108615353 time 739108354937 state 0x1 ref 1 00000100:00100000:1.0:1789760749.811441:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739108615353 time 739142372548 state 0x0 ref 1 00000100:00100000:1.0:1789760749.811453:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484255232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760749.811497:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22052864, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:1.0:1789760749.811505:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.811521:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:1.0:1789760749.811525:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760749.811531:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22052864->22056959] (req 22052864->22056959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.811545:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22052864->22056959] (req 22052864->22056959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760749.846612:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972702 is committed 00002000:00010000:3.0:1789760749.846727:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22052864 -> 22056960 00010000:00010000:3.0:1789760749.846742:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22052864->22056959] (req 22052864->22056959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.846755:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22052864->22056959] (req 22052864->22056959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.846765:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22052864->22056959] (req 22052864->22056959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.846773:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760749.846779:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22052864->22056959] (req 22052864->22056959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.846794:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22052864->22056959] (req 22052864->22056959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.846813:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22052864->22056959] (req 22052864->22056959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.846822:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208cf5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22052864->22056959] (req 22052864->22056959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760749.846916:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484255232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35459us (35658us total) trans 4294972702 rc 0/0 00000100:00100000:3.0:1789760749.846926:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5413 00000100:00100000:3.0:1789760749.846935:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739108615353 time 739142372548 state 0x2 ref 0 00000100:00100000:2.0:1789760749.850609:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760749.850694:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484255360 00000100:00100000:0.0:1789760749.850704:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484255360 00000100:00100000:0.0:1789760749.850760:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739108615353 time 739142372548 state 0x0 ref 1 00000100:00100000:0.0:1789760749.850775:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739142632964 time 739142372548 state 0x1 ref 1 00000100:00100000:0.0:1789760749.850805:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739142632964 time 739181736005 state 0x0 ref 1 00000100:00100000:0.0:1789760749.850817:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484255360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760749.850865:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22056960, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:0.0:1789760749.850874:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.850888:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:0.0:1789760749.850892:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760749.850899:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22056960->22061055] (req 22056960->22061055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.850913:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22056960->22061055] (req 22056960->22061055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760749.872002:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972703 is committed 00002000:00010000:0.0:1789760749.872173:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22056960 -> 22061056 00010000:00010000:0.0:1789760749.872187:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22056960->22061055] (req 22056960->22061055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.872204:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22056960->22061055] (req 22056960->22061055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.872212:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22056960->22061055] (req 22056960->22061055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.872219:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760749.872223:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22056960->22061055] (req 22056960->22061055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.872231:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22056960->22061055] (req 22056960->22061055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.872250:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22056960->22061055] (req 22056960->22061055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760749.872268:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208cfc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22056960->22061055] (req 22056960->22061055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760749.872362:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484255360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 21542us (21757us total) trans 4294972703 rc 0/0 00000100:00100000:0.0:1789760749.872372:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5414 00000100:00100000:0.0:1789760749.872381:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739142632964 time 739181736005 state 0x2 ref 0 00000100:00100000:0.0:1789760749.876113:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760749.876190:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484255488 00000100:00100000:2.0:1789760749.876200:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484255488 00000100:00100000:2.0:1789760749.876265:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739142632964 time 739181736005 state 0x0 ref 1 00000100:00100000:2.0:1789760749.876280:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739181996421 time 739181736005 state 0x1 ref 1 00000100:00100000:2.0:1789760749.876309:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739181996421 time 739207239654 state 0x0 ref 1 00000100:00100000:2.0:1789760749.876321:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484255488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760749.876376:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22061056, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:2.0:1789760749.876386:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.876401:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:2.0:1789760749.876406:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760749.876412:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22061056->22065151] (req 22061056->22065151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.876425:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22061056->22065151] (req 22061056->22065151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760749.911775:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22061056 -> 22065152 00010000:00010000:2.0:1789760749.911794:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22061056->22065151] (req 22061056->22065151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.911807:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22061056->22065151] (req 22061056->22065151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.911817:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22061056->22065151] (req 22061056->22065151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.911824:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760749.911829:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22061056->22065151] (req 22061056->22065151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.911838:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22061056->22065151] (req 22061056->22065151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.911856:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22061056->22065151] (req 22061056->22065151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760749.911864:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d03 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22061056->22065151] (req 22061056->22065151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760749.911944:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484255488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35620us (35835us total) trans 4294972704 rc 0/0 00000100:00100000:2.0:1789760749.911957:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5415 00000100:00100000:2.0:1789760749.911966:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739181996421 time 739207239654 state 0x2 ref 0 00000001:00080000:3.0:1789760749.914823:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972704 is committed 00000100:00100000:2.0:1789760749.914940:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760749.915028:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484255616 00000100:00100000:0.0:1789760749.915037:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484255616 00000100:00100000:1.0:1789760749.915430:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739181996421 time 739207239654 state 0x0 ref 1 00000100:00100000:1.0:1789760749.915446:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739207500070 time 739207239654 state 0x1 ref 1 00000100:00100000:1.0:1789760749.915511:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739207500070 time 739246440867 state 0x0 ref 1 00000100:00100000:1.0:1789760749.915528:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484255616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760749.915578:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22065152, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:1.0:1789760749.915587:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.915602:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:1.0:1789760749.915606:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760749.915613:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22065152->22069247] (req 22065152->22069247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.915626:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22065152->22069247] (req 22065152->22069247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760749.961675:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972705 is committed 00002000:00010000:1.0:1789760749.961769:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22065152 -> 22069248 00010000:00010000:1.0:1789760749.961783:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22065152->22069247] (req 22065152->22069247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.961795:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22065152->22069247] (req 22065152->22069247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.961805:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22065152->22069247] (req 22065152->22069247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.961812:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760749.961817:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22065152->22069247] (req 22065152->22069247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.961826:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22065152->22069247] (req 22065152->22069247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.961845:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22065152->22069247] (req 22065152->22069247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760749.961852:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d0a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22065152->22069247] (req 22065152->22069247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760749.961938:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484255616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46409us (46998us total) trans 4294972705 rc 0/0 00000100:00100000:1.0:1789760749.961947:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5416 00000100:00100000:1.0:1789760749.961956:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739207500070 time 739246440867 state 0x2 ref 0 00000100:00100000:2.0:1789760749.967121:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760749.967267:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484255744 00000100:00100000:3.0:1789760749.967277:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484255744 00000100:00100000:3.0:1789760749.967330:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739207500070 time 739246440867 state 0x0 ref 1 00000100:00100000:3.0:1789760749.967345:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739246701283 time 739246440867 state 0x1 ref 1 00000100:00100000:3.0:1789760749.967373:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739246701283 time 739298303682 state 0x0 ref 1 00000100:00100000:3.0:1789760749.967386:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484255744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760749.967441:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22069248, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:3.0:1789760749.967451:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.967471:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:3.0:1789760749.967475:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760749.967482:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22069248->22073343] (req 22069248->22073343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760749.967496:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22069248->22073343] (req 22069248->22073343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760750.003864:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972706 is committed 00002000:00010000:1.0:1789760750.003962:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22069248 -> 22073344 00010000:00010000:1.0:1789760750.003979:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22069248->22073343] (req 22069248->22073343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.003992:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22069248->22073343] (req 22069248->22073343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.004001:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22069248->22073343] (req 22069248->22073343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.004009:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760750.004014:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22069248->22073343] (req 22069248->22073343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.004023:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22069248->22073343] (req 22069248->22073343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.004241:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22069248->22073343] (req 22069248->22073343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.004258:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208d11 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22069248->22073343] (req 22069248->22073343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760750.004344:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484255744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36956us (37228us total) trans 4294972706 rc 0/0 00000100:00100000:1.0:1789760750.004354:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5417 00000100:00100000:1.0:1789760750.004365:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739246701283 time 739298303682 state 0x2 ref 0 00000100:00100000:0.0:1789760750.007978:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760750.008330:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484255872 00000100:00100000:1.0:1789760750.008340:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484255872 00000100:00100000:1.0:1789760750.008502:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739246701283 time 739298303682 state 0x0 ref 1 00000100:00100000:1.0:1789760750.008518:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739298564098 time 739298303682 state 0x1 ref 1 00000100:00100000:1.0:1789760750.008552:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739298564098 time 739339482566 state 0x0 ref 1 00000100:00100000:1.0:1789760750.008566:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484255872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760750.008614:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22073344, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:1.0:1789760750.008623:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.008639:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:1.0:1789760750.008644:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760750.008655:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22073344->22077439] (req 22073344->22077439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760750.008658:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00010000:00010000:1.0:1789760750.008669:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22073344->22077439] (req 22073344->22077439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760750.045541:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22073344 -> 22077440 00010000:00010000:1.0:1789760750.045560:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22073344->22077439] (req 22073344->22077439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.045572:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22073344->22077439] (req 22073344->22077439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.045582:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22073344->22077439] (req 22073344->22077439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.045590:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760750.045594:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22073344->22077439] (req 22073344->22077439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.045603:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22073344->22077439] (req 22073344->22077439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.045627:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22073344->22077439] (req 22073344->22077439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.045634:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d18 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22073344->22077439] (req 22073344->22077439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760750.045723:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484255872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37156us (37750us total) trans 4294972707 rc 0/0 00000100:00100000:1.0:1789760750.045734:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5418 00000100:00100000:1.0:1789760750.045743:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739298564098 time 739339482566 state 0x2 ref 0 00000001:00080000:3.0:1789760750.045818:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972707 is committed 00000100:00100000:3.0:1789760750.049651:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760750.050055:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484256000 00000100:00100000:2.0:1789760750.050092:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484256000 00000100:00100000:2.0:1789760750.050145:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739298564098 time 739339482566 state 0x0 ref 1 00000100:00100000:2.0:1789760750.050160:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739339742982 time 739339482566 state 0x1 ref 1 00000100:00100000:2.0:1789760750.050184:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739339742982 time 739381115665 state 0x0 ref 1 00000100:00100000:2.0:1789760750.050196:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484256000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760750.050245:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22077440, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:2.0:1789760750.050263:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.050279:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:2.0:1789760750.050284:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760750.050290:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22077440->22081535] (req 22077440->22081535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.050304:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22077440->22081535] (req 22077440->22081535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760750.093622:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972708 is committed 00002000:00010000:0.0:1789760750.093730:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22077440 -> 22081536 00010000:00010000:0.0:1789760750.093745:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22077440->22081535] (req 22077440->22081535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.093758:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22077440->22081535] (req 22077440->22081535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.093767:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22077440->22081535] (req 22077440->22081535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.093774:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760750.093780:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22077440->22081535] (req 22077440->22081535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.093789:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22077440->22081535] (req 22077440->22081535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.093809:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22077440->22081535] (req 22077440->22081535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.093817:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396208d1f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22077440->22081535] (req 22077440->22081535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760750.093917:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484256000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43717us (44270us total) trans 4294972708 rc 0/0 00000100:00100000:0.0:1789760750.093927:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5419 00000100:00100000:0.0:1789760750.093937:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739339742982 time 739381115665 state 0x2 ref 0 00000100:00100000:0.0:1789760750.098797:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760750.098875:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484256128 00000100:00100000:3.0:1789760750.098883:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484256128 00000100:00100000:3.0:1789760750.098930:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739339742982 time 739381115665 state 0x0 ref 1 00000100:00100000:3.0:1789760750.098945:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739381376081 time 739381115665 state 0x1 ref 1 00000100:00100000:3.0:1789760750.098974:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739381376081 time 739429905431 state 0x0 ref 1 00000100:00100000:3.0:1789760750.098986:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484256128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760750.099029:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22081536, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:3.0:1789760750.099047:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.099101:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:3.0:1789760750.099106:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760750.099113:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22081536->22085631] (req 22081536->22085631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.099127:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22081536->22085631] (req 22081536->22085631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760750.131787:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972709 is committed 00002000:00010000:0.0:1789760750.131895:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22081536 -> 22085632 00010000:00010000:0.0:1789760750.131910:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22081536->22085631] (req 22081536->22085631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.131923:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22081536->22085631] (req 22081536->22085631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.131933:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22081536->22085631] (req 22081536->22085631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.131940:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760750.131945:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22081536->22085631] (req 22081536->22085631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.131954:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22081536->22085631] (req 22081536->22085631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.131972:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22081536->22085631] (req 22081536->22085631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.131985:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208d26 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22081536->22085631] (req 22081536->22085631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760750.132109:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484256128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33120us (33317us total) trans 4294972709 rc 0/0 00000100:00100000:0.0:1789760750.132119:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5420 00000100:00100000:0.0:1789760750.132128:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739381376081 time 739429905431 state 0x2 ref 0 00000100:00100000:2.0:1789760750.138493:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760750.141560:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484256256 00000100:00100000:1.0:1789760750.141574:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484256256 00000100:00100000:0.0:1789760750.141810:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739381376081 time 739429905431 state 0x0 ref 1 00000100:00100000:0.0:1789760750.141826:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739430165847 time 739429905431 state 0x1 ref 1 00000100:00100000:0.0:1789760750.141908:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739430165847 time 739472838092 state 0x0 ref 1 00000100:00100000:0.0:1789760750.141924:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484256256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760750.141973:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22085632, mtime: 0x6aad94ed, atime: 0x0, ctime: 0x6aad94ed, blocks: 0xb007 00010000:00010000:0.0:1789760750.141982:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.141998:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:0.0:1789760750.142002:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760750.142009:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22085632->22089727] (req 22085632->22089727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.142021:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22085632->22089727] (req 22085632->22089727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760750.184741:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972710 is committed 00002000:00010000:0.0:1789760750.184969:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22085632 -> 22089728 00002000:00010000:0.0:1789760750.184974:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760749 -> 1789760750 00002000:00010000:0.0:1789760750.184976:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760749 -> 1789760750 00010000:00010000:0.0:1789760750.184989:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22085632->22089727] (req 22085632->22089727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.185002:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22085632->22089727] (req 22085632->22089727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.185012:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22085632->22089727] (req 22085632->22089727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.185021:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760750.185026:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22085632->22089727] (req 22085632->22089727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.185047:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22085632->22089727] (req 22085632->22089727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.185107:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22085632->22089727] (req 22085632->22089727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.185116:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d2d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22085632->22089727] (req 22085632->22089727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760750.185284:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484256256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43358us (46794us total) trans 4294972710 rc 0/0 00000100:00100000:1.0:1789760750.185295:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5421 00000100:00100000:1.0:1789760750.185306:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739430165847 time 739472838092 state 0x2 ref 0 00000100:00100000:3.0:1789760750.192239:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760750.192411:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484256384 00000100:00100000:1.0:1789760750.192420:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484256384 00000100:00100000:1.0:1789760750.192475:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739430165847 time 739472838092 state 0x0 ref 1 00000100:00100000:1.0:1789760750.192491:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739473098508 time 739472838092 state 0x1 ref 1 00000100:00100000:1.0:1789760750.192524:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739473098508 time 739523455050 state 0x0 ref 1 00000100:00100000:1.0:1789760750.192538:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484256384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760750.192587:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22089728, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:1.0:1789760750.192595:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.192610:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:1.0:1789760750.192615:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760750.192621:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22089728->22093823] (req 22089728->22093823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.192635:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22089728->22093823] (req 22089728->22093823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760750.244518:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972711 is committed 00002000:00010000:0.0:1789760750.244884:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22089728 -> 22093824 00010000:00010000:0.0:1789760750.244899:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22089728->22093823] (req 22089728->22093823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.244912:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22089728->22093823] (req 22089728->22093823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.244922:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22089728->22093823] (req 22089728->22093823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.244930:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760750.244934:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22089728->22093823] (req 22089728->22093823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.244943:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22089728->22093823] (req 22089728->22093823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.244963:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22089728->22093823] (req 22089728->22093823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.244971:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396208d34 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22089728->22093823] (req 22089728->22093823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760750.245108:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484256384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52568us (52873us total) trans 4294972711 rc 0/0 00000100:00100000:0.0:1789760750.245124:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5422 00000100:00100000:0.0:1789760750.245134:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739473098508 time 739523455050 state 0x2 ref 0 00000100:00100000:0.0:1789760750.252671:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760750.252860:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484256512 00000100:00100000:2.0:1789760750.252879:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484256512 00000100:00100000:2.0:1789760750.252932:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739473098508 time 739523455050 state 0x0 ref 1 00000100:00100000:2.0:1789760750.252946:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739523715466 time 739523455050 state 0x1 ref 1 00000100:00100000:2.0:1789760750.252977:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739523715466 time 739583907995 state 0x0 ref 1 00000100:00100000:2.0:1789760750.252996:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484256512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760750.253044:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22093824, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:2.0:1789760750.253054:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.253099:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:2.0:1789760750.253105:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760750.253112:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22093824->22097919] (req 22093824->22097919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.253126:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22093824->22097919] (req 22093824->22097919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760750.301790:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972712 is committed 00002000:00010000:2.0:1789760750.301921:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22093824 -> 22097920 00010000:00010000:2.0:1789760750.301938:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22093824->22097919] (req 22093824->22097919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.301950:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22093824->22097919] (req 22093824->22097919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.301959:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22093824->22097919] (req 22093824->22097919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.301967:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760750.301971:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22093824->22097919] (req 22093824->22097919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.301980:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22093824->22097919] (req 22093824->22097919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.302001:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22093824->22097919] (req 22093824->22097919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.302010:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208d3b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22093824->22097919] (req 22093824->22097919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760750.302334:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484256512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49341us (49668us total) trans 4294972712 rc 0/0 00000100:00100000:2.0:1789760750.302345:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5423 00000100:00100000:2.0:1789760750.302364:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739523715466 time 739583907995 state 0x2 ref 0 00000100:00100000:2.0:1789760750.308889:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760750.308959:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484256640 00000100:00100000:0.0:1789760750.308969:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484256640 00000100:00100000:0.0:1789760750.309025:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739523715466 time 739583907995 state 0x0 ref 1 00000100:00100000:0.0:1789760750.309041:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739584168411 time 739583907995 state 0x1 ref 1 00000100:00100000:0.0:1789760750.309161:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739584168411 time 739640090799 state 0x0 ref 1 00000100:00100000:0.0:1789760750.309177:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484256640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760750.309236:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22097920, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:0.0:1789760750.309245:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.309269:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:0.0:1789760750.309273:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760750.309280:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22097920->22102015] (req 22097920->22102015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.309297:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22097920->22102015] (req 22097920->22102015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760750.388302:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972713 is committed 00002000:00010000:0.0:1789760750.388488:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22097920 -> 22102016 00010000:00010000:0.0:1789760750.388503:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22097920->22102015] (req 22097920->22102015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.388516:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22097920->22102015] (req 22097920->22102015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.388525:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22097920->22102015] (req 22097920->22102015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.388533:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760750.388538:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22097920->22102015] (req 22097920->22102015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.388547:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22097920->22102015] (req 22097920->22102015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.388566:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22097920->22102015] (req 22097920->22102015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.388574:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d42 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22097920->22102015] (req 22097920->22102015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760750.388669:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484256640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79491us (79784us total) trans 4294972713 rc 0/0 00000100:00100000:0.0:1789760750.388682:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5424 00000100:00100000:0.0:1789760750.388691:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739584168411 time 739640090799 state 0x2 ref 0 00000100:00100000:3.0:1789760750.394212:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760750.394352:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484256768 00000100:00100000:1.0:1789760750.394363:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484256768 00000100:00100000:1.0:1789760750.394422:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739584168411 time 739640090799 state 0x0 ref 1 00000100:00100000:1.0:1789760750.394438:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739640351215 time 739640090799 state 0x1 ref 1 00000100:00100000:1.0:1789760750.394472:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739640351215 time 739725402547 state 0x0 ref 1 00000100:00100000:1.0:1789760750.394485:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484256768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760750.394537:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22102016, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:1.0:1789760750.394546:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.394563:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:1.0:1789760750.394567:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760750.394573:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22102016->22106111] (req 22102016->22106111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.394585:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22102016->22106111] (req 22102016->22106111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760750.440091:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22102016 -> 22106112 00010000:00010000:0.0:1789760750.440112:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22102016->22106111] (req 22102016->22106111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.440126:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22102016->22106111] (req 22102016->22106111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.440136:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22102016->22106111] (req 22102016->22106111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.440143:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760750.440148:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22102016->22106111] (req 22102016->22106111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.440157:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22102016->22106111] (req 22102016->22106111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.440177:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22102016->22106111] (req 22102016->22106111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.440185:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396208d49 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22102016->22106111] (req 22102016->22106111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760750.440277:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484256768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45789us (46068us total) trans 4294972714 rc 0/0 00000100:00100000:0.0:1789760750.440288:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5425 00000100:00100000:0.0:1789760750.440299:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739640351215 time 739725402547 state 0x2 ref 0 00000001:00080000:1.0:1789760750.441642:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972714 is committed 00000100:00100000:1.0:1789760750.451701:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760750.451806:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484256896 00000100:00100000:0.0:1789760750.451817:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484256896 00000100:00100000:0.0:1789760750.451882:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739640351215 time 739725402547 state 0x0 ref 1 00000100:00100000:0.0:1789760750.451898:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739725662963 time 739725402547 state 0x1 ref 1 00000100:00100000:0.0:1789760750.451928:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739725662963 time 739782859050 state 0x0 ref 1 00000100:00100000:0.0:1789760750.451941:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484256896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760750.451992:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22106112, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:0.0:1789760750.452001:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.452017:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:0.0:1789760750.452021:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760750.452027:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22106112->22110207] (req 22106112->22110207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.452053:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22106112->22110207] (req 22106112->22110207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.480895:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760750.504872:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972715 is committed 00002000:00010000:3.0:1789760750.505191:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22106112 -> 22110208 00010000:00010000:3.0:1789760750.505207:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22106112->22110207] (req 22106112->22110207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.505221:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22106112->22110207] (req 22106112->22110207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.505231:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22106112->22110207] (req 22106112->22110207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.505239:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760750.505244:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22106112->22110207] (req 22106112->22110207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.505265:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22106112->22110207] (req 22106112->22110207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.505286:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22106112->22110207] (req 22106112->22110207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.505295:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208d50 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22106112->22110207] (req 22106112->22110207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760750.505458:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484256896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53514us (53762us total) trans 4294972715 rc 0/0 00000100:00100000:3.0:1789760750.505469:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5426 00000100:00100000:3.0:1789760750.505480:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739725662963 time 739782859050 state 0x2 ref 0 00000100:00100000:2.0:1789760750.511786:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760750.511878:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484257024 00000100:00100000:0.0:1789760750.511889:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484257024 00000100:00100000:0.0:1789760750.511946:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739725662963 time 739782859050 state 0x0 ref 1 00000100:00100000:0.0:1789760750.511966:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739783119466 time 739782859050 state 0x1 ref 1 00000100:00100000:0.0:1789760750.511997:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739783119466 time 739842928405 state 0x0 ref 1 00000100:00100000:0.0:1789760750.512011:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484257024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760750.512093:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22110208, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:0.0:1789760750.512102:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.512130:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:0.0:1789760750.512135:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760750.512141:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22110208->22114303] (req 22110208->22114303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.512157:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22110208->22114303] (req 22110208->22114303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760750.554335:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972716 is committed 00002000:00010000:0.0:1789760750.554506:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22110208 -> 22114304 00010000:00010000:0.0:1789760750.554522:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22110208->22114303] (req 22110208->22114303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.554534:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22110208->22114303] (req 22110208->22114303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.554544:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22110208->22114303] (req 22110208->22114303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.554556:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760750.554561:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22110208->22114303] (req 22110208->22114303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.554570:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22110208->22114303] (req 22110208->22114303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.554591:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22110208->22114303] (req 22110208->22114303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760750.554598:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208d57 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22110208->22114303] (req 22110208->22114303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760750.554711:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484257024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42698us (42930us total) trans 4294972716 rc 0/0 00000100:00100000:0.0:1789760750.554722:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5427 00000100:00100000:0.0:1789760750.554734:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739783119466 time 739842928405 state 0x2 ref 0 00000100:00100000:0.0:1789760750.558621:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760750.558693:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484257152 00000100:00100000:3.0:1789760750.558703:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484257152 00000100:00100000:3.0:1789760750.558754:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739783119466 time 739842928405 state 0x0 ref 1 00000100:00100000:3.0:1789760750.558769:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739843188821 time 739842928405 state 0x1 ref 1 00000100:00100000:3.0:1789760750.558802:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739843188821 time 739889733242 state 0x0 ref 1 00000100:00100000:3.0:1789760750.558814:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484257152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760750.558861:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22114304, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:3.0:1789760750.558870:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.558886:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:3.0:1789760750.558891:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760750.558897:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22114304->22118399] (req 22114304->22118399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.558912:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22114304->22118399] (req 22114304->22118399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760750.602120:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972717 is committed 00002000:00010000:1.0:1789760750.602528:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22114304 -> 22118400 00010000:00010000:1.0:1789760750.602544:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22114304->22118399] (req 22114304->22118399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.602557:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22114304->22118399] (req 22114304->22118399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.602567:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22114304->22118399] (req 22114304->22118399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.602575:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760750.602580:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22114304->22118399] (req 22114304->22118399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.602593:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22114304->22118399] (req 22114304->22118399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.602613:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22114304->22118399] (req 22114304->22118399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.602620:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208d5e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22114304->22118399] (req 22114304->22118399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760750.602714:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484257152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43897us (44098us total) trans 4294972717 rc 0/0 00000100:00100000:1.0:1789760750.602725:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5428 00000100:00100000:1.0:1789760750.602742:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739843188821 time 739889733242 state 0x2 ref 0 00000100:00100000:1.0:1789760750.608220:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760750.608306:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484257280 00000100:00100000:2.0:1789760750.608317:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484257280 00000100:00100000:2.0:1789760750.608371:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739843188821 time 739889733242 state 0x0 ref 1 00000100:00100000:2.0:1789760750.608387:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739889993658 time 739889733242 state 0x1 ref 1 00000100:00100000:2.0:1789760750.608455:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739889993658 time 739939385263 state 0x0 ref 1 00000100:00100000:2.0:1789760750.608469:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484257280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760750.608526:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22118400, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:2.0:1789760750.608536:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.608552:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:2.0:1789760750.608556:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760750.608563:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22118400->22122495] (req 22118400->22122495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.608577:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22118400->22122495] (req 22118400->22122495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760750.650850:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972718 is committed 00002000:00010000:2.0:1789760750.650922:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22118400 -> 22122496 00010000:00010000:2.0:1789760750.650937:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22118400->22122495] (req 22118400->22122495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.650949:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22118400->22122495] (req 22118400->22122495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.650959:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22118400->22122495] (req 22118400->22122495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.650967:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760750.650972:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22118400->22122495] (req 22118400->22122495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.650982:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22118400->22122495] (req 22118400->22122495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.651002:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22118400->22122495] (req 22118400->22122495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.651010:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d65 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22118400->22122495] (req 22118400->22122495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760750.651219:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484257280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42747us (43003us total) trans 4294972718 rc 0/0 00000100:00100000:2.0:1789760750.651229:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5429 00000100:00100000:2.0:1789760750.651239:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739889993658 time 739939385263 state 0x2 ref 0 00000100:00100000:2.0:1789760750.655501:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760750.655590:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484257408 00000100:00100000:3.0:1789760750.655600:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484257408 00000100:00100000:3.0:1789760750.655653:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739889993658 time 739939385263 state 0x0 ref 1 00000100:00100000:3.0:1789760750.655667:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739939645679 time 739939385263 state 0x1 ref 1 00000100:00100000:3.0:1789760750.655709:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739939645679 time 739986640143 state 0x0 ref 1 00000100:00100000:3.0:1789760750.655721:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484257408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760750.655765:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22122496, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:3.0:1789760750.655774:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.655793:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=739s v=(0 0) 00010000:00010000:3.0:1789760750.655798:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760750.655805:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22122496->22126591] (req 22122496->22126591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.655819:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22122496->22126591] (req 22122496->22126591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760750.690385:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972719 is committed 00002000:00010000:3.0:1789760750.690483:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22122496 -> 22126592 00010000:00010000:3.0:1789760750.690499:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22122496->22126591] (req 22122496->22126591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.690511:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22122496->22126591] (req 22122496->22126591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.690519:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22122496->22126591] (req 22122496->22126591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.690526:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760750.690531:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22122496->22126591] (req 22122496->22126591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.690540:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22122496->22126591] (req 22122496->22126591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.690563:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22122496->22126591] (req 22122496->22126591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.690572:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d6c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22122496->22126591] (req 22122496->22126591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760750.690691:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484257408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34967us (35196us total) trans 4294972719 rc 0/0 00000100:00100000:3.0:1789760750.690704:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5430 00000100:00100000:3.0:1789760750.690714:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739939645679 time 739986640143 state 0x2 ref 0 00000100:00100000:0.0:1789760750.699918:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760750.700024:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484257536 00000100:00100000:2.0:1789760750.700034:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484257536 00000100:00100000:2.0:1789760750.700180:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739939645679 time 739986640143 state 0x0 ref 1 00000100:00100000:2.0:1789760750.700196:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739986900559 time 739986640143 state 0x1 ref 1 00000100:00100000:2.0:1789760750.700220:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739986900559 time 740031150864 state 0x0 ref 1 00000100:00100000:2.0:1789760750.700232:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484257536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760750.700302:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22126592, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:2.0:1789760750.700312:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.700327:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:2.0:1789760750.700332:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760750.700404:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22126592->22130687] (req 22126592->22130687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.700419:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22126592->22130687] (req 22126592->22130687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760750.729894:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972720 is committed 00002000:00010000:2.0:1789760750.729966:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22126592 -> 22130688 00010000:00010000:2.0:1789760750.729980:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22126592->22130687] (req 22126592->22130687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.729998:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22126592->22130687] (req 22126592->22130687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.730009:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22126592->22130687] (req 22126592->22130687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.730016:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760750.730021:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22126592->22130687] (req 22126592->22130687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.730220:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22126592->22130687] (req 22126592->22130687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.730241:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22126592->22130687] (req 22126592->22130687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.730260:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d73 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22126592->22130687] (req 22126592->22130687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760750.730470:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484257536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30234us (30556us total) trans 4294972720 rc 0/0 00000100:00100000:2.0:1789760750.730481:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5431 00000100:00100000:2.0:1789760750.730490:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739986900559 time 740031150864 state 0x2 ref 0 00000100:00100000:1.0:1789760750.733512:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760750.733598:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484257664 00000100:00100000:3.0:1789760750.733608:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484257664 00000100:00100000:3.0:1789760750.733660:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 739986900559 time 740031150864 state 0x0 ref 1 00000100:00100000:3.0:1789760750.733674:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740031411280 time 740031150864 state 0x1 ref 1 00000100:00100000:3.0:1789760750.733739:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740031411280 time 740064669848 state 0x0 ref 1 00000100:00100000:3.0:1789760750.733752:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484257664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760750.733799:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22130688, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:3.0:1789760750.733808:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.733825:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:3.0:1789760750.733829:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760750.733836:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22130688->22134783] (req 22130688->22134783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.733849:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22130688->22134783] (req 22130688->22134783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760750.772422:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972721 is committed 00002000:00010000:1.0:1789760750.772598:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22130688 -> 22134784 00010000:00010000:1.0:1789760750.772614:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22130688->22134783] (req 22130688->22134783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.772628:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22130688->22134783] (req 22130688->22134783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.772637:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22130688->22134783] (req 22130688->22134783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.772645:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760750.772650:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22130688->22134783] (req 22130688->22134783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.772659:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22130688->22134783] (req 22130688->22134783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.772678:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22130688->22134783] (req 22130688->22134783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.772686:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208d7a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22130688->22134783] (req 22130688->22134783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760750.772869:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484257664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39114us (39362us total) trans 4294972721 rc 0/0 00000100:00100000:1.0:1789760750.772878:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5432 00000100:00100000:1.0:1789760750.772887:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740031411280 time 740064669848 state 0x2 ref 0 00000100:00100000:0.0:1789760750.777699:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760750.777896:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484257792 00000100:00100000:2.0:1789760750.777906:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484257792 00000100:00100000:2.0:1789760750.777955:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740031411280 time 740064669848 state 0x0 ref 1 00000100:00100000:2.0:1789760750.777970:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740064930264 time 740064669848 state 0x1 ref 1 00000100:00100000:2.0:1789760750.778048:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740064930264 time 740108978827 state 0x0 ref 1 00000100:00100000:2.0:1789760750.778061:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484257792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760750.778290:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22134784, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:2.0:1789760750.778301:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.778318:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:2.0:1789760750.778323:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760750.778329:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22134784->22138879] (req 22134784->22138879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.778344:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22134784->22138879] (req 22134784->22138879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760750.815769:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972722 is committed 00002000:00010000:1.0:1789760750.816163:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22134784 -> 22138880 00010000:00010000:1.0:1789760750.816180:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22134784->22138879] (req 22134784->22138879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.816192:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22134784->22138879] (req 22134784->22138879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.816206:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22134784->22138879] (req 22134784->22138879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.816214:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760750.816219:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22134784->22138879] (req 22134784->22138879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.816228:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22134784->22138879] (req 22134784->22138879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.816263:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22134784->22138879] (req 22134784->22138879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.816272:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208d81 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22134784->22138879] (req 22134784->22138879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760750.817266:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484257792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39187us (39555us total) trans 4294972722 rc 0/0 00000100:00100000:1.0:1789760750.817278:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5433 00000100:00100000:1.0:1789760750.817288:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740064930264 time 740108978827 state 0x2 ref 0 00000100:00100000:0.0:1789760750.822574:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760750.822777:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484257920 00000100:00100000:2.0:1789760750.822787:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484257920 00000100:00100000:2.0:1789760750.822839:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740064930264 time 740108978827 state 0x0 ref 1 00000100:00100000:2.0:1789760750.822855:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740109239243 time 740108978827 state 0x1 ref 1 00000100:00100000:2.0:1789760750.822932:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740109239243 time 740153861265 state 0x0 ref 1 00000100:00100000:2.0:1789760750.822948:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484257920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760750.823009:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22138880, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:2.0:1789760750.823018:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.823043:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:2.0:1789760750.823048:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760750.823053:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22138880->22142975] (req 22138880->22142975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.823085:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22138880->22142975] (req 22138880->22142975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760750.859302:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972723 is committed 00002000:00010000:2.0:1789760750.859650:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22138880 -> 22142976 00010000:00010000:2.0:1789760750.859665:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22138880->22142975] (req 22138880->22142975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.859677:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22138880->22142975] (req 22138880->22142975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.859686:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22138880->22142975] (req 22138880->22142975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.859694:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760750.859699:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22138880->22142975] (req 22138880->22142975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.859708:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22138880->22142975] (req 22138880->22142975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.859726:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22138880->22142975] (req 22138880->22142975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.859733:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208d88 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22138880->22142975] (req 22138880->22142975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760750.859826:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484257920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36875us (37257us total) trans 4294972723 rc 0/0 00000100:00100000:2.0:1789760750.859838:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5434 00000100:00100000:2.0:1789760750.859848:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740109239243 time 740153861265 state 0x2 ref 0 00000100:00100000:0.0:1789760750.863239:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760750.863591:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484258048 00000100:00100000:2.0:1789760750.863602:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484258048 00000100:00100000:2.0:1789760750.863656:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740109239243 time 740153861265 state 0x0 ref 1 00000100:00100000:2.0:1789760750.863672:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740154121681 time 740153861265 state 0x1 ref 1 00000100:00100000:2.0:1789760750.863702:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740154121681 time 740194633702 state 0x0 ref 1 00000100:00100000:2.0:1789760750.863715:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760750.863764:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22142976, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:2.0:1789760750.863774:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.863791:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:2.0:1789760750.863795:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760750.863802:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22142976->22147071] (req 22142976->22147071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.863816:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22142976->22147071] (req 22142976->22147071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760750.896137:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972724 is committed 00002000:00010000:3.0:1789760750.896443:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22142976 -> 22147072 00010000:00010000:3.0:1789760750.896459:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22142976->22147071] (req 22142976->22147071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.896471:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22142976->22147071] (req 22142976->22147071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.896480:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22142976->22147071] (req 22142976->22147071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.896487:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760750.896492:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22142976->22147071] (req 22142976->22147071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.896500:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22142976->22147071] (req 22142976->22147071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.896519:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22142976->22147071] (req 22142976->22147071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760750.896527:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208d8f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22142976->22147071] (req 22142976->22147071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760750.896652:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32934us (33418us total) trans 4294972724 rc 0/0 00000100:00100000:3.0:1789760750.896662:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5435 00000100:00100000:3.0:1789760750.896672:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740154121681 time 740194633702 state 0x2 ref 0 00000100:00100000:0.0:1789760750.900023:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760750.900253:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484258176 00000100:00100000:1.0:1789760750.900263:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484258176 00000100:00100000:1.0:1789760750.900321:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740154121681 time 740194633702 state 0x0 ref 1 00000100:00100000:1.0:1789760750.900403:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740194894118 time 740194633702 state 0x1 ref 1 00000100:00100000:1.0:1789760750.900470:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740194894118 time 740231400428 state 0x0 ref 1 00000100:00100000:1.0:1789760750.900483:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760750.900533:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22147072, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:1.0:1789760750.900540:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.900553:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:1.0:1789760750.900557:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760750.900561:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22147072->22151167] (req 22147072->22151167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.900572:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22147072->22151167] (req 22147072->22151167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760750.936142:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972725 is committed 00002000:00010000:1.0:1789760750.936237:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22147072 -> 22151168 00010000:00010000:1.0:1789760750.936261:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22147072->22151167] (req 22147072->22151167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.936274:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22147072->22151167] (req 22147072->22151167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.936283:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22147072->22151167] (req 22147072->22151167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.936295:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760750.936300:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22147072->22151167] (req 22147072->22151167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.936308:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22147072->22151167] (req 22147072->22151167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.936327:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22147072->22151167] (req 22147072->22151167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.936334:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d96 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22147072->22151167] (req 22147072->22151167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760750.936426:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35941us (36408us total) trans 4294972725 rc 0/0 00000100:00100000:1.0:1789760750.936436:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5436 00000100:00100000:1.0:1789760750.936445:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740194894118 time 740231400428 state 0x2 ref 0 00000100:00100000:0.0:1789760750.943526:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760750.943606:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484258304 00000100:00100000:1.0:1789760750.943616:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484258304 00000100:00100000:1.0:1789760750.943670:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740194894118 time 740231400428 state 0x0 ref 1 00000100:00100000:1.0:1789760750.943685:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740231660844 time 740231400428 state 0x1 ref 1 00000100:00100000:1.0:1789760750.943713:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740231660844 time 740274644912 state 0x0 ref 1 00000100:00100000:1.0:1789760750.943729:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760750.943776:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22151168, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:1.0:1789760750.943785:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.943802:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:1.0:1789760750.943807:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760750.943814:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22151168->22155263] (req 22151168->22155263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.943827:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22151168->22155263] (req 22151168->22155263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760750.988928:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22151168 -> 22155264 00010000:00010000:1.0:1789760750.988948:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22151168->22155263] (req 22151168->22155263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.988963:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22151168->22155263] (req 22151168->22155263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.988973:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22151168->22155263] (req 22151168->22155263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.988980:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760750.988985:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22151168->22155263] (req 22151168->22155263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.988996:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22151168->22155263] (req 22151168->22155263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.989018:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22151168->22155263] (req 22151168->22155263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760750.989038:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396208d9d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22151168->22155263] (req 22151168->22155263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760750.989166:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484258304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45434us (45645us total) trans 4294972726 rc 0/0 00000100:00100000:1.0:1789760750.989176:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5437 00000100:00100000:1.0:1789760750.989187:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740231660844 time 740274644912 state 0x2 ref 0 00000001:00080000:3.0:1789760750.989326:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972726 is committed 00000100:00100000:0.0:1789760750.995259:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760750.999341:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484258432 00000100:00100000:2.0:1789760750.999358:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484258432 00000100:00100000:2.0:1789760750.999428:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740231660844 time 740274644912 state 0x0 ref 1 00000100:00100000:2.0:1789760750.999445:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740274905328 time 740274644912 state 0x1 ref 1 00000100:00100000:2.0:1789760750.999484:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740274905328 time 740330414403 state 0x0 ref 1 00000100:00100000:2.0:1789760750.999499:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760750.999550:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22155264, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:2.0:1789760750.999560:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.999575:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:2.0:1789760750.999580:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760750.999586:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22155264->22159359] (req 22155264->22159359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760750.999606:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22155264->22159359] (req 22155264->22159359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760751.064684:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972727 is committed 00002000:00010000:2.0:1789760751.064858:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22155264 -> 22159360 00010000:00010000:2.0:1789760751.064875:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22155264->22159359] (req 22155264->22159359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.064888:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22155264->22159359] (req 22155264->22159359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.064898:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22155264->22159359] (req 22155264->22159359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.064905:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760751.064910:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22155264->22159359] (req 22155264->22159359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.064919:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22155264->22159359] (req 22155264->22159359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.064940:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22155264->22159359] (req 22155264->22159359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.064947:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208da4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22155264->22159359] (req 22155264->22159359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760751.065551:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66051us (70305us total) trans 4294972727 rc 0/0 00000100:00100000:2.0:1789760751.065562:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5438 00000100:00100000:2.0:1789760751.065572:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740274905328 time 740330414403 state 0x2 ref 0 00000100:00100000:0.0:1789760751.072657:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760751.072750:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484258560 00000100:00100000:3.0:1789760751.072760:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484258560 00000100:00100000:3.0:1789760751.072817:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740274905328 time 740330414403 state 0x0 ref 1 00000100:00100000:3.0:1789760751.072832:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740330674819 time 740330414403 state 0x1 ref 1 00000100:00100000:3.0:1789760751.072866:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740330674819 time 740403796979 state 0x0 ref 1 00000100:00100000:3.0:1789760751.072880:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760751.072932:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22159360, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:3.0:1789760751.072941:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.072958:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:3.0:1789760751.072962:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760751.072968:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22159360->22163455] (req 22159360->22163455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.072983:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22159360->22163455] (req 22159360->22163455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760751.109739:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22159360 -> 22163456 00010000:00010000:3.0:1789760751.109758:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22159360->22163455] (req 22159360->22163455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.109778:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22159360->22163455] (req 22159360->22163455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.109788:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22159360->22163455] (req 22159360->22163455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.109795:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760751.109800:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22159360->22163455] (req 22159360->22163455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.109809:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22159360->22163455] (req 22159360->22163455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.109828:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22159360->22163455] (req 22159360->22163455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.109835:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dab lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22159360->22163455] (req 22159360->22163455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760751.109942:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484258560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37060us (37289us total) trans 4294972728 rc 0/0 00000100:00100000:3.0:1789760751.109953:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5439 00000100:00100000:3.0:1789760751.109962:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740330674819 time 740403796979 state 0x2 ref 0 00000001:00080000:1.0:1789760751.110589:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972728 is committed 00000100:00100000:1.0:1789760751.116708:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760751.116800:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484258688 00000100:00100000:3.0:1789760751.116810:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484258688 00000100:00100000:3.0:1789760751.116875:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740330674819 time 740403796979 state 0x0 ref 1 00000100:00100000:3.0:1789760751.116890:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740404057395 time 740403796979 state 0x1 ref 1 00000100:00100000:3.0:1789760751.116921:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740404057395 time 740447852025 state 0x0 ref 1 00000100:00100000:3.0:1789760751.116933:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706e850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484258688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760751.116983:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22163456, mtime: 0x6aad94ee, atime: 0x0, ctime: 0x6aad94ee, blocks: 0xb007 00010000:00010000:3.0:1789760751.116992:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.117008:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:3.0:1789760751.117012:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760751.117019:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22163456->22167551] (req 22163456->22167551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.117034:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22163456->22167551] (req 22163456->22167551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760751.150419:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972729 is committed 00002000:00010000:3.0:1789760751.150521:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22163456 -> 22167552 00002000:00010000:3.0:1789760751.150526:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760750 -> 1789760751 00002000:00010000:3.0:1789760751.150528:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760750 -> 1789760751 00010000:00010000:3.0:1789760751.150540:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22163456->22167551] (req 22163456->22167551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.150553:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22163456->22167551] (req 22163456->22167551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.150562:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22163456->22167551] (req 22163456->22167551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.150569:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760751.150573:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22163456->22167551] (req 22163456->22167551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.150581:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22163456->22167551] (req 22163456->22167551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.150599:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22163456->22167551] (req 22163456->22167551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.150606:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208db2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22163456->22167551] (req 22163456->22167551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760751.150791:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706e850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484258688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33855us (34087us total) trans 4294972729 rc 0/0 00000100:00100000:3.0:1789760751.150802:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5440 00000100:00100000:3.0:1789760751.150811:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740404057395 time 740447852025 state 0x2 ref 0 00000100:00100000:2.0:1789760751.161451:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760751.162577:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484258816 00000100:00100000:1.0:1789760751.162589:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484258816 00000100:00100000:1.0:1789760751.162660:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740404057395 time 740447852025 state 0x0 ref 1 00000100:00100000:1.0:1789760751.162677:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740448112441 time 740447852025 state 0x1 ref 1 00000100:00100000:1.0:1789760751.162719:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740448112441 time 740493649907 state 0x0 ref 1 00000100:00100000:1.0:1789760751.162734:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760751.162783:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22167552, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:1.0:1789760751.162793:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.162808:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:1.0:1789760751.162812:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760751.162817:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22167552->22171647] (req 22167552->22171647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.162830:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22167552->22171647] (req 22167552->22171647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760751.204270:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972730 is committed 00002000:00010000:1.0:1789760751.204541:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22167552 -> 22171648 00010000:00010000:1.0:1789760751.204557:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22167552->22171647] (req 22167552->22171647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.204571:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22167552->22171647] (req 22167552->22171647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.204582:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22167552->22171647] (req 22167552->22171647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.204590:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.204595:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22167552->22171647] (req 22167552->22171647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.204605:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22167552->22171647] (req 22167552->22171647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.204626:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22167552->22171647] (req 22167552->22171647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.204635:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208db9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22167552->22171647] (req 22167552->22171647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760751.204867:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42130us (43419us total) trans 4294972730 rc 0/0 00000100:00100000:1.0:1789760751.204878:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5441 00000100:00100000:1.0:1789760751.204890:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740448112441 time 740493649907 state 0x2 ref 0 00000100:00100000:0.0:1789760751.211762:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760751.211859:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484258944 00000100:00100000:2.0:1789760751.211869:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484258944 00000100:00100000:2.0:1789760751.211925:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740448112441 time 740493649907 state 0x0 ref 1 00000100:00100000:2.0:1789760751.211939:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740493910323 time 740493649907 state 0x1 ref 1 00000100:00100000:2.0:1789760751.211970:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740493910323 time 740542901032 state 0x0 ref 1 00000100:00100000:2.0:1789760751.211983:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760751.212044:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22171648, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:2.0:1789760751.212053:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.212099:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:2.0:1789760751.212104:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760751.212110:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22171648->22175743] (req 22171648->22175743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.212124:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22171648->22175743] (req 22171648->22175743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760751.253317:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972731 is committed 00002000:00010000:1.0:1789760751.253603:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22171648 -> 22175744 00010000:00010000:1.0:1789760751.253619:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22171648->22175743] (req 22171648->22175743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.253633:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22171648->22175743] (req 22171648->22175743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.253654:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22171648->22175743] (req 22171648->22175743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.253662:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.253666:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22171648->22175743] (req 22171648->22175743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.253681:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22171648->22175743] (req 22171648->22175743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.253701:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22171648->22175743] (req 22171648->22175743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.253709:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208dc0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22171648->22175743] (req 22171648->22175743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760751.253799:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484258944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41813us (42041us total) trans 4294972731 rc 0/0 00000100:00100000:1.0:1789760751.254520:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5442 00000100:00100000:1.0:1789760751.254531:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740493910323 time 740542901032 state 0x2 ref 0 00000100:00100000:0.0:1789760751.259436:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760751.259555:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484259072 00000100:00100000:3.0:1789760751.259564:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484259072 00000100:00100000:3.0:1789760751.259618:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740493910323 time 740542901032 state 0x0 ref 1 00000100:00100000:3.0:1789760751.259633:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740543161448 time 740542901032 state 0x1 ref 1 00000100:00100000:3.0:1789760751.259672:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740543161448 time 740590602647 state 0x0 ref 1 00000100:00100000:3.0:1789760751.259684:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484259072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760751.259733:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22175744, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:3.0:1789760751.259741:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.259757:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:3.0:1789760751.259761:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760751.259767:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22175744->22179839] (req 22175744->22179839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.259781:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22175744->22179839] (req 22175744->22179839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760751.313547:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972732 is committed 00002000:00010000:0.0:1789760751.313642:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22175744 -> 22179840 00010000:00010000:0.0:1789760751.313656:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22175744->22179839] (req 22175744->22179839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.313665:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22175744->22179839] (req 22175744->22179839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.313672:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22175744->22179839] (req 22175744->22179839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.313676:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760751.313679:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22175744->22179839] (req 22175744->22179839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.313688:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22175744->22179839] (req 22175744->22179839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.313709:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22175744->22179839] (req 22175744->22179839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.313718:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208dc7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22175744->22179839] (req 22175744->22179839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760751.313923:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484259072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54235us (54491us total) trans 4294972732 rc 0/0 00000100:00100000:0.0:1789760751.313933:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5443 00000100:00100000:0.0:1789760751.313943:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740543161448 time 740590602647 state 0x2 ref 0 00000100:00100000:2.0:1789760751.317540:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760751.317644:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484259200 00000100:00100000:3.0:1789760751.317654:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484259200 00000100:00100000:3.0:1789760751.317713:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740543161448 time 740590602647 state 0x0 ref 1 00000100:00100000:3.0:1789760751.317729:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740590863063 time 740590602647 state 0x1 ref 1 00000100:00100000:3.0:1789760751.317817:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740590863063 time 740648746296 state 0x0 ref 1 00000100:00100000:3.0:1789760751.317830:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484259200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760751.317878:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22179840, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:3.0:1789760751.317886:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.317901:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:3.0:1789760751.317905:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760751.317912:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22179840->22183935] (req 22179840->22183935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.317926:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22179840->22183935] (req 22179840->22183935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760751.365314:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972733 is committed 00002000:00010000:1.0:1789760751.366517:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22179840 -> 22183936 00010000:00010000:1.0:1789760751.366534:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22179840->22183935] (req 22179840->22183935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.366548:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22179840->22183935] (req 22179840->22183935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.366556:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22179840->22183935] (req 22179840->22183935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.366567:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.366572:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22179840->22183935] (req 22179840->22183935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.366580:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22179840->22183935] (req 22179840->22183935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.366599:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22179840->22183935] (req 22179840->22183935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.366607:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dce lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22179840->22183935] (req 22179840->22183935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760751.366746:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484259200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48913us (49213us total) trans 4294972733 rc 0/0 00000100:00100000:1.0:1789760751.366756:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5444 00000100:00100000:1.0:1789760751.366766:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740590863063 time 740648746296 state 0x2 ref 0 00000100:00100000:2.0:1789760751.370393:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760751.370465:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484259328 00000100:00100000:1.0:1789760751.370475:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484259328 00000100:00100000:1.0:1789760751.370527:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740590863063 time 740648746296 state 0x0 ref 1 00000100:00100000:1.0:1789760751.370543:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740649006712 time 740648746296 state 0x1 ref 1 00000100:00100000:1.0:1789760751.370574:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740649006712 time 740701504225 state 0x0 ref 1 00000100:00100000:1.0:1789760751.370590:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484259328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760751.370638:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22183936, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:1.0:1789760751.370647:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.370664:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:1.0:1789760751.370668:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760751.370676:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22183936->22188031] (req 22183936->22188031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.370691:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22183936->22188031] (req 22183936->22188031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760751.407112:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972734 is committed 00002000:00010000:1.0:1789760751.407467:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22183936 -> 22188032 00010000:00010000:1.0:1789760751.407481:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22183936->22188031] (req 22183936->22188031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.407494:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22183936->22188031] (req 22183936->22188031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.407504:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22183936->22188031] (req 22183936->22188031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.407511:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.407516:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22183936->22188031] (req 22183936->22188031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.407524:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22183936->22188031] (req 22183936->22188031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.407543:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22183936->22188031] (req 22183936->22188031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.407552:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208dd5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22183936->22188031] (req 22183936->22188031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760751.407718:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484259328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37126us (37330us total) trans 4294972734 rc 0/0 00000100:00100000:1.0:1789760751.407728:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5445 00000100:00100000:1.0:1789760751.407737:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740649006712 time 740701504225 state 0x2 ref 0 00000100:00100000:1.0:1789760751.413280:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760751.413835:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484259456 00000100:00100000:3.0:1789760751.413846:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484259456 00000100:00100000:3.0:1789760751.413897:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740649006712 time 740701504225 state 0x0 ref 1 00000100:00100000:3.0:1789760751.413913:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740701764641 time 740701504225 state 0x1 ref 1 00000100:00100000:3.0:1789760751.413943:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740701764641 time 740744874448 state 0x0 ref 1 00000100:00100000:3.0:1789760751.413956:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484259456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760751.414006:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22188032, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:3.0:1789760751.414014:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.414040:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:3.0:1789760751.414045:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760751.414052:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22188032->22192127] (req 22188032->22192127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.414094:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22188032->22192127] (req 22188032->22192127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760751.471901:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22188032 -> 22192128 00010000:00010000:3.0:1789760751.471920:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22188032->22192127] (req 22188032->22192127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.471933:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22188032->22192127] (req 22188032->22192127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.471941:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22188032->22192127] (req 22188032->22192127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.471948:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760751.471952:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22188032->22192127] (req 22188032->22192127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.471961:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22188032->22192127] (req 22188032->22192127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.471979:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22188032->22192127] (req 22188032->22192127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.471986:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208ddc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22188032->22192127] (req 22188032->22192127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760751.472123:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484259456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58164us (58846us total) trans 4294972735 rc 0/0 00000100:00100000:3.0:1789760751.472133:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5446 00000100:00100000:3.0:1789760751.472142:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740701764641 time 740744874448 state 0x2 ref 0 00000001:00080000:1.0:1789760751.473264:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972735 is committed 00000100:00100000:3.0:1789760751.476895:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760751.477023:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484259584 00000100:00100000:0.0:1789760751.477033:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484259584 00000100:00100000:0.0:1789760751.477129:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740701764641 time 740744874448 state 0x0 ref 1 00000100:00100000:0.0:1789760751.477144:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740745134864 time 740744874448 state 0x1 ref 1 00000100:00100000:0.0:1789760751.477166:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740745134864 time 740808097757 state 0x0 ref 1 00000100:00100000:0.0:1789760751.477179:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defde300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484259584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760751.477226:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22192128, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:0.0:1789760751.477235:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.477260:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:0.0:1789760751.477265:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760751.477271:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22192128->22196223] (req 22192128->22196223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.477284:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22192128->22196223] (req 22192128->22196223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.500261:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760751.512052:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972736 is committed 00002000:00010000:0.0:1789760751.512228:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22192128 -> 22196224 00010000:00010000:0.0:1789760751.512252:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22192128->22196223] (req 22192128->22196223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.512266:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22192128->22196223] (req 22192128->22196223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.512275:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22192128->22196223] (req 22192128->22196223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.512282:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760751.512287:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22192128->22196223] (req 22192128->22196223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.512296:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22192128->22196223] (req 22192128->22196223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.512317:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22192128->22196223] (req 22192128->22196223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.512324:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208de3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22192128->22196223] (req 22192128->22196223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760751.512530:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defde300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484259584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35349us (35638us total) trans 4294972736 rc 0/0 00000100:00100000:0.0:1789760751.512543:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5447 00000100:00100000:0.0:1789760751.512552:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740745134864 time 740808097757 state 0x2 ref 0 00000100:00100000:2.0:1789760751.518197:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760751.518409:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484259712 00000100:00100000:3.0:1789760751.518420:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484259712 00000100:00100000:3.0:1789760751.518473:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740745134864 time 740808097757 state 0x0 ref 1 00000100:00100000:3.0:1789760751.518489:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740808358173 time 740808097757 state 0x1 ref 1 00000100:00100000:3.0:1789760751.518519:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740808358173 time 740849450252 state 0x0 ref 1 00000100:00100000:3.0:1789760751.518532:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484259712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760751.518581:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22196224, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:3.0:1789760751.518589:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.518605:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:3.0:1789760751.518609:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760751.518615:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22196224->22200319] (req 22196224->22200319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.518629:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22196224->22200319] (req 22196224->22200319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760751.556464:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22196224 -> 22200320 00010000:00010000:3.0:1789760751.556484:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22196224->22200319] (req 22196224->22200319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.556499:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22196224->22200319] (req 22196224->22200319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.556508:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22196224->22200319] (req 22196224->22200319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.556515:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760751.556520:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22196224->22200319] (req 22196224->22200319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.556530:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22196224->22200319] (req 22196224->22200319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.556551:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22196224->22200319] (req 22196224->22200319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.556560:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208dea lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22196224->22200319] (req 22196224->22200319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760751.556665:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484259712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38130us (38473us total) trans 4294972737 rc 0/0 00000100:00100000:3.0:1789760751.556681:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5448 00000100:00100000:3.0:1789760751.556691:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740808358173 time 740849450252 state 0x2 ref 0 00000001:00080000:1.0:1789760751.557666:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972737 is committed 00000100:00100000:1.0:1789760751.562021:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760751.562998:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484259840 00000100:00100000:2.0:1789760751.563010:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484259840 00000100:00100000:2.0:1789760751.563119:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740808358173 time 740849450252 state 0x0 ref 1 00000100:00100000:2.0:1789760751.563133:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740849710668 time 740849450252 state 0x1 ref 1 00000100:00100000:2.0:1789760751.563154:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740849710668 time 740894085411 state 0x0 ref 1 00000100:00100000:2.0:1789760751.563168:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484259840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760751.563217:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22200320, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:2.0:1789760751.563226:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.563250:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:2.0:1789760751.563254:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760751.563261:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22200320->22204415] (req 22200320->22204415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.563275:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22200320->22204415] (req 22200320->22204415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760751.605874:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972738 is committed 00002000:00010000:1.0:1789760751.606156:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22200320 -> 22204416 00010000:00010000:1.0:1789760751.606171:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22200320->22204415] (req 22200320->22204415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.606185:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22200320->22204415] (req 22200320->22204415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.606200:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22200320->22204415] (req 22200320->22204415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.606208:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.606217:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22200320->22204415] (req 22200320->22204415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.606226:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22200320->22204415] (req 22200320->22204415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.606254:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22200320->22204415] (req 22200320->22204415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.606261:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396208df1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22200320->22204415] (req 22200320->22204415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760751.606349:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484259840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43180us (44334us total) trans 4294972738 rc 0/0 00000100:00100000:1.0:1789760751.606360:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5449 00000100:00100000:1.0:1789760751.606369:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740849710668 time 740894085411 state 0x2 ref 0 00000100:00100000:2.0:1789760751.609580:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760751.609679:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484259968 00000100:00100000:1.0:1789760751.609689:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484259968 00000100:00100000:1.0:1789760751.609742:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740849710668 time 740894085411 state 0x0 ref 1 00000100:00100000:1.0:1789760751.609759:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740894345827 time 740894085411 state 0x1 ref 1 00000100:00100000:1.0:1789760751.609892:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740894345827 time 740940821985 state 0x0 ref 1 00000100:00100000:1.0:1789760751.609906:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484259968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760751.609954:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22204416, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:1.0:1789760751.609963:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.609978:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:1.0:1789760751.609982:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760751.609988:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22204416->22208511] (req 22204416->22208511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.610000:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22204416->22208511] (req 22204416->22208511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760751.647527:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972739 is committed 00002000:00010000:1.0:1789760751.647618:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22204416 -> 22208512 00010000:00010000:1.0:1789760751.647634:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22204416->22208511] (req 22204416->22208511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.647647:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22204416->22208511] (req 22204416->22208511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.647656:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22204416->22208511] (req 22204416->22208511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.647663:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.647669:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22204416->22208511] (req 22204416->22208511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.647678:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22204416->22208511] (req 22204416->22208511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.647698:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22204416->22208511] (req 22204416->22208511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.647705:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208df8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22204416->22208511] (req 22204416->22208511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760751.647801:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484259968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37892us (38226us total) trans 4294972739 rc 0/0 00000100:00100000:1.0:1789760751.647811:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5450 00000100:00100000:1.0:1789760751.647821:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740894345827 time 740940821985 state 0x2 ref 0 00000100:00100000:2.0:1789760751.654649:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760751.655822:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484260096 00000100:00100000:0.0:1789760751.655834:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484260096 00000100:00100000:0.0:1789760751.655898:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740894345827 time 740940821985 state 0x0 ref 1 00000100:00100000:0.0:1789760751.655914:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740941082401 time 740940821985 state 0x1 ref 1 00000100:00100000:0.0:1789760751.655935:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740941082401 time 740986866389 state 0x0 ref 1 00000100:00100000:0.0:1789760751.655949:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760751.655997:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22208512, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:0.0:1789760751.656006:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.656023:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=740s v=(0 0) 00010000:00010000:0.0:1789760751.656027:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760751.656033:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22208512->22212607] (req 22208512->22212607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.656048:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22208512->22212607] (req 22208512->22212607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760751.716870:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972740 is committed 00002000:00010000:3.0:1789760751.716972:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22208512 -> 22212608 00010000:00010000:3.0:1789760751.716987:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22208512->22212607] (req 22208512->22212607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.716999:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22208512->22212607] (req 22208512->22212607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.717008:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22208512->22212607] (req 22208512->22212607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.717014:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760751.717019:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22208512->22212607] (req 22208512->22212607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.717038:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22208512->22212607] (req 22208512->22212607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.717061:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22208512->22212607] (req 22208512->22212607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.717100:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208dff lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22208512->22212607] (req 22208512->22212607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760751.717196:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61246us (62552us total) trans 4294972740 rc 0/0 00000100:00100000:3.0:1789760751.717205:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5451 00000100:00100000:3.0:1789760751.717215:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740941082401 time 740986866389 state 0x2 ref 0 00000100:00100000:2.0:1789760751.720341:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760751.720418:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484260224 00000100:00100000:3.0:1789760751.720428:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484260224 00000100:00100000:3.0:1789760751.720482:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740941082401 time 740986866389 state 0x0 ref 1 00000100:00100000:3.0:1789760751.720499:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740987126805 time 740986866389 state 0x1 ref 1 00000100:00100000:3.0:1789760751.720528:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740987126805 time 741051458479 state 0x0 ref 1 00000100:00100000:3.0:1789760751.720540:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484260224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760751.720590:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22212608, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:3.0:1789760751.720598:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.720624:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:3.0:1789760751.720629:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760751.720635:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22212608->22216703] (req 22212608->22216703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.720650:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22212608->22216703] (req 22212608->22216703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760751.745682:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972741 is committed 00002000:00010000:3.0:1789760751.745892:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22212608 -> 22216704 00010000:00010000:3.0:1789760751.745909:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22212608->22216703] (req 22212608->22216703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.745921:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22212608->22216703] (req 22212608->22216703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.745930:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22212608->22216703] (req 22212608->22216703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.745937:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760751.745942:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22212608->22216703] (req 22212608->22216703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.745951:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22212608->22216703] (req 22212608->22216703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.745971:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22212608->22216703] (req 22212608->22216703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.745979:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e06 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22212608->22216703] (req 22212608->22216703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760751.746196:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484260224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25652us (25859us total) trans 4294972741 rc 0/0 00000100:00100000:3.0:1789760751.746210:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5452 00000100:00100000:3.0:1789760751.746220:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740987126805 time 741051458479 state 0x2 ref 0 00000100:00100000:1.0:1789760751.753194:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760751.753394:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484260352 00000100:00100000:2.0:1789760751.753403:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484260352 00000100:00100000:2.0:1789760751.753458:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 740987126805 time 741051458479 state 0x0 ref 1 00000100:00100000:2.0:1789760751.753473:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741051718895 time 741051458479 state 0x1 ref 1 00000100:00100000:2.0:1789760751.753505:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741051718895 time 741084435637 state 0x0 ref 1 00000100:00100000:2.0:1789760751.753517:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760751.753568:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22216704, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:2.0:1789760751.753577:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.753593:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:2.0:1789760751.753598:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760751.753603:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22216704->22220799] (req 22216704->22220799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760751.753616:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22216704->22220799] (req 22216704->22220799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760751.783727:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972742 is committed 00002000:00010000:1.0:1789760751.783818:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22216704 -> 22220800 00010000:00010000:1.0:1789760751.783834:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22216704->22220799] (req 22216704->22220799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.783846:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22216704->22220799] (req 22216704->22220799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.783861:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22216704->22220799] (req 22216704->22220799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.783869:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.783874:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22216704->22220799] (req 22216704->22220799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.783883:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22216704->22220799] (req 22216704->22220799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.783903:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22216704->22220799] (req 22216704->22220799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.783911:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208e0d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22216704->22220799] (req 22216704->22220799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760751.784054:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30533us (30863us total) trans 4294972742 rc 0/0 00000100:00100000:1.0:1789760751.784113:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5453 00000100:00100000:1.0:1789760751.784123:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741051718895 time 741084435637 state 0x2 ref 0 00000100:00100000:0.0:1789760751.788272:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760751.788365:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484260480 00000100:00100000:1.0:1789760751.788375:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484260480 00000100:00100000:1.0:1789760751.788432:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741051718895 time 741084435637 state 0x0 ref 1 00000100:00100000:1.0:1789760751.788447:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741084696053 time 741084435637 state 0x1 ref 1 00000100:00100000:1.0:1789760751.788505:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741084696053 time 741119435622 state 0x0 ref 1 00000100:00100000:1.0:1789760751.788517:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760751.788561:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22220800, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:1.0:1789760751.788570:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.788585:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:1.0:1789760751.788589:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760751.788594:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22220800->22224895] (req 22220800->22224895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.788608:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22220800->22224895] (req 22220800->22224895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760751.833385:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22220800 -> 22224896 00010000:00010000:1.0:1789760751.833408:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22220800->22224895] (req 22220800->22224895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.833421:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22220800->22224895] (req 22220800->22224895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.833432:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22220800->22224895] (req 22220800->22224895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.833439:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.833444:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22220800->22224895] (req 22220800->22224895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.833453:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22220800->22224895] (req 22220800->22224895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.833472:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22220800->22224895] (req 22220800->22224895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.833480:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e14 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22220800->22224895] (req 22220800->22224895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760751.833922:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972743 is committed 00000100:00100000:1.0:1789760751.833990:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45462us (45715us total) trans 4294972743 rc 0/0 00000100:00100000:1.0:1789760751.834001:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5454 00000100:00100000:1.0:1789760751.834012:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741084696053 time 741119435622 state 0x2 ref 0 00000100:00100000:2.0:1789760751.841250:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760751.841337:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484260608 00000100:00100000:3.0:1789760751.841346:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484260608 00000100:00100000:3.0:1789760751.841397:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741084696053 time 741119435622 state 0x0 ref 1 00000100:00100000:3.0:1789760751.841411:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741119696038 time 741119435622 state 0x1 ref 1 00000100:00100000:3.0:1789760751.841442:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741119696038 time 741172373136 state 0x0 ref 1 00000100:00100000:3.0:1789760751.841454:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484260608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760751.841504:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22224896, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:3.0:1789760751.841513:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.841528:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:3.0:1789760751.841532:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760751.841539:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22224896->22228991] (req 22224896->22228991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.841552:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22224896->22228991] (req 22224896->22228991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760751.890643:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972744 is committed 00002000:00010000:1.0:1789760751.890737:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22224896 -> 22228992 00010000:00010000:1.0:1789760751.890753:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22224896->22228991] (req 22224896->22228991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.890767:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22224896->22228991] (req 22224896->22228991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.890777:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22224896->22228991] (req 22224896->22228991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.890791:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.890796:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22224896->22228991] (req 22224896->22228991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.890806:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22224896->22228991] (req 22224896->22228991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.890827:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22224896->22228991] (req 22224896->22228991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.890835:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396208e1b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22224896->22228991] (req 22224896->22228991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760751.890967:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484260608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49510us (49733us total) trans 4294972744 rc 0/0 00000100:00100000:1.0:1789760751.891178:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5455 00000100:00100000:1.0:1789760751.891189:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741119696038 time 741172373136 state 0x2 ref 0 00000100:00100000:2.0:1789760751.894941:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760751.897113:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484260736 00000100:00100000:3.0:1789760751.897127:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484260736 00000100:00100000:3.0:1789760751.897188:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741119696038 time 741172373136 state 0x0 ref 1 00000100:00100000:3.0:1789760751.897204:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741172633552 time 741172373136 state 0x1 ref 1 00000100:00100000:3.0:1789760751.897299:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741172633552 time 741228229010 state 0x0 ref 1 00000100:00100000:3.0:1789760751.897313:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abd180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760751.897362:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22228992, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:3.0:1789760751.897371:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.897386:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:3.0:1789760751.897390:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760751.897397:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22228992->22233087] (req 22228992->22233087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.897410:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22228992->22233087] (req 22228992->22233087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760751.936974:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972745 is committed 00002000:00010000:3.0:1789760751.937112:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22228992 -> 22233088 00010000:00010000:3.0:1789760751.937127:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22228992->22233087] (req 22228992->22233087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.937144:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22228992->22233087] (req 22228992->22233087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.937153:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22228992->22233087] (req 22228992->22233087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.937160:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760751.937165:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22228992->22233087] (req 22228992->22233087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.937174:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22228992->22233087] (req 22228992->22233087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.937193:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22228992->22233087] (req 22228992->22233087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760751.937200:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208e22 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22228992->22233087] (req 22228992->22233087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760751.937482:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abd180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40165us (42543us total) trans 4294972745 rc 0/0 00000100:00100000:3.0:1789760751.937492:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5456 00000100:00100000:3.0:1789760751.937502:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741172633552 time 741228229010 state 0x2 ref 0 00000100:00100000:0.0:1789760751.940881:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760751.940984:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484260864 00000100:00100000:1.0:1789760751.940995:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484260864 00000100:00100000:1.0:1789760751.941187:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741172633552 time 741228229010 state 0x0 ref 1 00000100:00100000:1.0:1789760751.941202:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741228489426 time 741228229010 state 0x1 ref 1 00000100:00100000:1.0:1789760751.941280:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741228489426 time 741272210629 state 0x0 ref 1 00000100:00100000:1.0:1789760751.941295:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484260864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760751.941546:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22233088, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:1.0:1789760751.941556:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.941572:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:1.0:1789760751.941576:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760751.941582:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22233088->22237183] (req 22233088->22237183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.941597:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22233088->22237183] (req 22233088->22237183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760751.982840:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972746 is committed 00002000:00010000:1.0:1789760751.983268:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22233088 -> 22237184 00010000:00010000:1.0:1789760751.983283:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22233088->22237183] (req 22233088->22237183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.983297:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22233088->22237183] (req 22233088->22237183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.983307:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22233088->22237183] (req 22233088->22237183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.983315:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760751.983321:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22233088->22237183] (req 22233088->22237183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.983409:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22233088->22237183] (req 22233088->22237183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.983433:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22233088->22237183] (req 22233088->22237183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760751.983441:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e29 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22233088->22237183] (req 22233088->22237183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760751.983538:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484260864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42242us (42663us total) trans 4294972746 rc 0/0 00000100:00100000:1.0:1789760751.983549:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5457 00000100:00100000:1.0:1789760751.983558:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741228489426 time 741272210629 state 0x2 ref 0 00000100:00100000:3.0:1789760751.987155:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760751.987228:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484260992 00000100:00100000:0.0:1789760751.987248:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484260992 00000100:00100000:0.0:1789760751.987305:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741228489426 time 741272210629 state 0x0 ref 1 00000100:00100000:0.0:1789760751.987320:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741272471045 time 741272210629 state 0x1 ref 1 00000100:00100000:0.0:1789760751.987357:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741272471045 time 741318288018 state 0x0 ref 1 00000100:00100000:0.0:1789760751.987371:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defded80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760751.987422:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22237184, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:0.0:1789760751.987431:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.987446:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:0.0:1789760751.987451:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760751.987457:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22237184->22241279] (req 22237184->22241279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760751.987471:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22237184->22241279] (req 22237184->22241279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.012785:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c4578e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508924928:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760752.012852:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760752.012879:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508924928 00000100:00100000:2.0:1789760752.012887:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508924928 00000100:00100000:2.0:1789760752.012953:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 388 00000100:00100000:2.0:1789760752.012962:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508924928:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:2.0:1789760752.012997:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760752.013041:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508924928:12345-0@lo:13:osp-pre-1-1.0 Request processed in 79us (188us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760752.013049:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 388 00000100:00080000:2.0:1789760752.013103:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:0.0:1789760752.013134:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c4578e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508924928:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760752.013271:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1789760752.013279:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c4578e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508925056:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760752.013306:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760752.014169:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508925056 00000100:00100000:1.0:1789760752.014174:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508925056 00000100:00100000:1.0:1789760752.014212:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 389 00000100:00100000:1.0:1789760752.014219:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508925056:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:1.0:1789760752.014234:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760752.014281:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508925056:12345-0@lo:13:osp-pre-1-0.0 Request processed in 63us (974us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760752.014288:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 389 00000100:00100000:1.0:1789760752.014310:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508925184:0@lo:400:kworker.0 00000100:00100000:1.0:1789760752.014463:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760752.014490:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e87100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508925696:0@lo:400:kworker.0 00000100:00100000:1.0:1789760752.014517:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760752.014536:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508926208:0@lo:400:kworker.0 00000100:00100000:1.0:1789760752.014557:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760752.014575:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1789760752.014580:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508925312:0@lo:400:kworker.0 00000100:00100000:1.0:1789760752.014602:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760752.014620:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508925824:0@lo:400:kworker.0 00000100:00100000:1.0:1789760752.014638:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760752.014654:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508926336:0@lo:400:kworker.0 00000100:00100000:1.0:1789760752.014673:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760752.014702:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508925184 00000100:00100000:1.0:1789760752.014707:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508925184 00000100:00100000:1.0:1789760752.014736:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 482 00000100:00100000:1.0:1789760752.014742:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b7850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508925184:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760752.014776:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b7850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508925184:12345-0@lo:400:kworker.0 Request processed in 33us (312us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760752.014783:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 482 00000100:00100000:2.0:1789760752.014799:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508925184:0@lo:400:kworker.0 00000100:00100000:1.0:1789760752.014808:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508925696 00000100:00100000:1.0:1789760752.014812:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508925696 00000100:00100000:1.0:1789760752.015099:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 447 00000100:00100000:1.0:1789760752.015107:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508925696:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760752.015145:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508925696:12345-0@lo:400:kworker.0 Request processed in 39us (627us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760752.015151:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 447 00000100:00100000:3.0:1789760752.015208:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e87100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508925696:0@lo:400:kworker.0 00000100:00100000:2.0:1789760752.020332:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508926208 00000100:00100000:2.0:1789760752.020346:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508926208 00000100:00100000:0.0:1789760752.020347:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e87b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508925440:0@lo:400:kworker.0 00000100:00100000:2.0:1789760752.020409:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 327 00000100:00100000:2.0:1789760752.020419:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508926208:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.020421:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760752.020454:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e85880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508925952:0@lo:400:kworker.0 00000100:00100000:2.0:1789760752.020475:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508926208:12345-0@lo:400:kworker.0 Request processed in 58us (5916us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.020478:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760752.020483:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 327 00000100:00100000:0.0:1789760752.020494:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501b81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508926464:0@lo:400:kworker.0 00000100:00100000:2.0:1789760752.020507:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508925568:0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.020514:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760752.020531:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760752.020547:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e85500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508926080:0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.020553:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c4578e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508925056:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760752.020566:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760752.020582:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508925440 00000100:00100000:0.0:1789760752.020585:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508925440 00000100:00100000:2.0:1789760752.020589:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508925312 00000100:00100000:2.0:1789760752.020592:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508925312 00000100:00100000:2.0:1789760752.020608:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508925824 00000100:00100000:2.0:1789760752.020611:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508925824 00000100:00100000:0.0:1789760752.020612:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 448 00000100:00100000:0.0:1789760752.020618:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508925440:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760752.020623:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508926336 00000100:00100000:2.0:1789760752.020625:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508926336 00000100:00100000:2.0:1789760752.020636:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508925952 00000100:00100000:2.0:1789760752.020638:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508925952 00000100:00100000:2.0:1789760752.020649:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508926464 00000100:00100000:2.0:1789760752.020651:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508926464 00000100:00100000:0.0:1789760752.020653:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508925440:12345-0@lo:400:kworker.0 Request processed in 36us (231us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.020660:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 448 00000100:00100000:2.0:1789760752.020661:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508925568 00000100:00100000:2.0:1789760752.020662:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508925568 00000100:00100000:2.0:1789760752.020673:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 935 00000100:00100000:2.0:1789760752.020695:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508925312:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760752.020725:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508925312:12345-0@lo:400:kworker.0 Request processed in 32us (6122us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760752.020880:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508926080 00000100:00100000:3.0:1789760752.020885:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508926080 00000100:00100000:3.0:1789760752.020917:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 328 00000100:00100000:3.0:1789760752.020923:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abfb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508926080:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760752.020960:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abfb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508926080:12345-0@lo:400:kworker.0 Request processed in 37us (392us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760752.020965:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 328 00000100:00100000:3.0:1789760752.020993:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508925312:0@lo:400:kworker.0 00000100:00100000:3.0:1789760752.021004:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508926208:0@lo:400:kworker.0 00000100:00100000:3.0:1789760752.021042:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e85500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508926080:0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021261:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 936 00000100:00100000:0.0:1789760752.021310:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 937 00000100:00100000:0.0:1789760752.021320:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508926336:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021432:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508926336:12345-0@lo:400:kworker.0 Request processed in 115us (6757us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.021439:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 937 00000100:00100000:0.0:1789760752.021456:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 938 00000100:00100000:0.0:1789760752.021464:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508925952:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021493:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508925952:12345-0@lo:400:kworker.0 Request processed in 31us (1014us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.021500:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 938 00000100:00100000:0.0:1789760752.021511:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 939 00000100:00100000:0.0:1789760752.021519:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508926464:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021548:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457ad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508926464:12345-0@lo:400:kworker.0 Request processed in 31us (1032us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.021554:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 939 00000100:00100000:0.0:1789760752.021566:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 940 00000100:00100000:0.0:1789760752.021573:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abed80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508925568:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021604:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abed80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508925568:12345-0@lo:400:kworker.0 Request processed in 32us (1071us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.021609:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 940 00000100:00100000:0.0:1789760752.021642:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508926336:0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021667:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508925568:0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021684:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508925824:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021720:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508925824:12345-0@lo:400:kworker.0 Request processed in 452us (7080us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.021726:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 936 00000100:00100000:0.0:1789760752.021753:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508925824:0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021778:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e87b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508925440:0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021788:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e85880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508925952:0@lo:400:kworker.0 00000100:00100000:0.0:1789760752.021797:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501b81f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508926464:0@lo:400:kworker.0 00000001:00080000:3.0:1789760752.022057:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972747 is committed 00002000:00010000:0.0:1789760752.022264:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22237184 -> 22241280 00010000:00010000:0.0:1789760752.022278:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22237184->22241279] (req 22237184->22241279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.022290:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22237184->22241279] (req 22237184->22241279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.022298:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22237184->22241279] (req 22237184->22241279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.022305:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.022309:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22237184->22241279] (req 22237184->22241279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.022317:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22237184->22241279] (req 22237184->22241279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.022553:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22237184->22241279] (req 22237184->22241279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.022563:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e30 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22237184->22241279] (req 22237184->22241279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760752.022603:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 935 00000100:00080000:2.0:1789760752.022633:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:0.0:1789760752.022636:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defded80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484260992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35266us (35487us total) trans 4294972747 rc 0/0 00000100:00080000:2.0:1789760752.022638:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:2.0:1789760752.022641:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:2.0:1789760752.022643:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:0.0:1789760752.022648:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5458 00000100:00100000:0.0:1789760752.022657:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741272471045 time 741318288018 state 0x2 ref 0 00000100:00100000:3.0:1789760752.026644:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.026737:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484261120 00000100:00100000:0.0:1789760752.026747:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484261120 00000100:00100000:0.0:1789760752.026823:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741272471045 time 741318288018 state 0x0 ref 1 00000100:00100000:0.0:1789760752.026839:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741318548434 time 741318288018 state 0x1 ref 1 00000100:00100000:0.0:1789760752.026872:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741318548434 time 741357803251 state 0x0 ref 1 00000100:00100000:0.0:1789760752.026886:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484261120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760752.026932:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22241280, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:0.0:1789760752.026943:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.026959:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:0.0:1789760752.026964:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760752.026970:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22241280->22245375] (req 22241280->22245375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.026984:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22241280->22245375] (req 22241280->22245375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760752.027214:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:3.0:1789760752.067031:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22241280 -> 22245376 00010000:00010000:3.0:1789760752.067050:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22241280->22245375] (req 22241280->22245375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.067091:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22241280->22245375] (req 22241280->22245375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.067101:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22241280->22245375] (req 22241280->22245375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.067109:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760752.067114:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22241280->22245375] (req 22241280->22245375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.067124:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22241280->22245375] (req 22241280->22245375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.067143:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22241280->22245375] (req 22241280->22245375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.067150:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208e37 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22241280->22245375] (req 22241280->22245375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760752.067243:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484261120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40348us (40596us total) trans 4294972748 rc 0/0 00000100:00100000:3.0:1789760752.067254:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5459 00000100:00100000:3.0:1789760752.067264:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741318548434 time 741357803251 state 0x2 ref 0 00000001:00080000:3.0:1789760752.067303:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972748 is committed 00000100:00100000:3.0:1789760752.070980:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760752.072839:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484261248 00000100:00100000:2.0:1789760752.072854:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484261248 00000100:00100000:0.0:1789760752.073041:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741318548434 time 741357803251 state 0x0 ref 1 00000100:00100000:0.0:1789760752.073061:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741358063667 time 741357803251 state 0x1 ref 1 00000100:00100000:0.0:1789760752.073425:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741358063667 time 741404324416 state 0x0 ref 1 00000100:00100000:0.0:1789760752.073445:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484261248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760752.073492:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22245376, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:0.0:1789760752.073500:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.073515:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:0.0:1789760752.073519:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760752.073526:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22245376->22249471] (req 22245376->22249471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.073539:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22245376->22249471] (req 22245376->22249471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760752.101805:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972749 is committed 00002000:00010000:0.0:1789760752.101907:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22245376 -> 22249472 00010000:00010000:0.0:1789760752.101921:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22245376->22249471] (req 22245376->22249471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.101936:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22245376->22249471] (req 22245376->22249471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.101945:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22245376->22249471] (req 22245376->22249471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.101956:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.101962:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22245376->22249471] (req 22245376->22249471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.101971:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22245376->22249471] (req 22245376->22249471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.101989:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22245376->22249471] (req 22245376->22249471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.101997:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e3e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22245376->22249471] (req 22245376->22249471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.102123:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484261248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28679us (31148us total) trans 4294972749 rc 0/0 00000100:00100000:0.0:1789760752.102133:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5460 00000100:00100000:0.0:1789760752.102142:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741358063667 time 741404324416 state 0x2 ref 0 00000100:00100000:2.0:1789760752.108392:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760752.108899:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484261376 00000100:00100000:1.0:1789760752.108910:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484261376 00000100:00100000:1.0:1789760752.108964:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741358063667 time 741404324416 state 0x0 ref 1 00000100:00100000:1.0:1789760752.108979:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741404584832 time 741404324416 state 0x1 ref 1 00000100:00100000:1.0:1789760752.109168:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741404584832 time 741440097668 state 0x0 ref 1 00000100:00100000:1.0:1789760752.109184:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484261376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760752.109246:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22249472, mtime: 0x6aad94ef, atime: 0x0, ctime: 0x6aad94ef, blocks: 0xb007 00010000:00010000:1.0:1789760752.109256:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.109273:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:1.0:1789760752.109278:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760752.109285:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22249472->22253567] (req 22249472->22253567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.109299:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22249472->22253567] (req 22249472->22253567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760752.156869:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972750 is committed 00002000:00010000:1.0:1789760752.156991:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22249472 -> 22253568 00002000:00010000:1.0:1789760752.156996:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760751 -> 1789760752 00002000:00010000:1.0:1789760752.156999:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760751 -> 1789760752 00010000:00010000:1.0:1789760752.157011:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22249472->22253567] (req 22249472->22253567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.157036:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22249472->22253567] (req 22249472->22253567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.157052:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22249472->22253567] (req 22249472->22253567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.157060:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760752.157094:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22249472->22253567] (req 22249472->22253567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.157105:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22249472->22253567] (req 22249472->22253567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.157125:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22249472->22253567] (req 22249472->22253567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.157134:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396208e45 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22249472->22253567] (req 22249472->22253567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760752.157217:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484261376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48033us (48890us total) trans 4294972750 rc 0/0 00000100:00100000:1.0:1789760752.157227:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5461 00000100:00100000:1.0:1789760752.157251:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741404584832 time 741440097668 state 0x2 ref 0 00000100:00100000:2.0:1789760752.161291:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.161448:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484261504 00000100:00100000:0.0:1789760752.161458:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484261504 00000100:00100000:0.0:1789760752.161520:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741404584832 time 741440097668 state 0x0 ref 1 00000100:00100000:0.0:1789760752.161536:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741440358084 time 741440097668 state 0x1 ref 1 00000100:00100000:0.0:1789760752.161598:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741440358084 time 741492528482 state 0x0 ref 1 00000100:00100000:0.0:1789760752.161615:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484261504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760752.161661:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22253568, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:0.0:1789760752.161669:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.161696:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:0.0:1789760752.161700:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760752.161706:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22253568->22257663] (req 22253568->22257663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.161720:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22253568->22257663] (req 22253568->22257663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760752.186269:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22253568 -> 22257664 00010000:00010000:2.0:1789760752.186290:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22253568->22257663] (req 22253568->22257663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.186303:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22253568->22257663] (req 22253568->22257663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.186317:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22253568->22257663] (req 22253568->22257663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.186457:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760752.186463:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22253568->22257663] (req 22253568->22257663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.186470:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22253568->22257663] (req 22253568->22257663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.186487:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22253568->22257663] (req 22253568->22257663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.186492:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e4c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22253568->22257663] (req 22253568->22257663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760752.186582:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484261504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24964us (25294us total) trans 4294972751 rc 0/0 00000100:00100000:2.0:1789760752.186592:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5462 00000100:00100000:2.0:1789760752.186602:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741440358084 time 741492528482 state 0x2 ref 0 00000001:00080000:2.0:1789760752.186640:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972751 is committed 00000100:00100000:2.0:1789760752.189175:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760752.189639:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484261632 00000100:00100000:3.0:1789760752.189650:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484261632 00000100:00100000:3.0:1789760752.189702:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741440358084 time 741492528482 state 0x0 ref 1 00000100:00100000:3.0:1789760752.189718:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741492788898 time 741492528482 state 0x1 ref 1 00000100:00100000:3.0:1789760752.189751:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741492788898 time 741520681385 state 0x0 ref 1 00000100:00100000:3.0:1789760752.189764:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484261632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760752.189810:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22257664, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:3.0:1789760752.189819:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.189833:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:3.0:1789760752.189838:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760752.189844:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22257664->22261759] (req 22257664->22261759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.189858:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22257664->22261759] (req 22257664->22261759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760752.222366:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22257664 -> 22261760 00010000:00010000:0.0:1789760752.222389:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22257664->22261759] (req 22257664->22261759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.222402:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22257664->22261759] (req 22257664->22261759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.222411:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22257664->22261759] (req 22257664->22261759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.222417:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.222421:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22257664->22261759] (req 22257664->22261759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.222430:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22257664->22261759] (req 22257664->22261759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.222449:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22257664->22261759] (req 22257664->22261759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.222457:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208e53 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22257664->22261759] (req 22257664->22261759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.222560:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484261632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32794us (33389us total) trans 4294972752 rc 0/0 00000100:00100000:0.0:1789760752.222570:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5463 00000100:00100000:0.0:1789760752.222580:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741492788898 time 741520681385 state 0x2 ref 0 00000001:00080000:2.0:1789760752.223421:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972752 is committed 00000100:00100000:3.0:1789760752.227282:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760752.227447:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484261760 00000100:00100000:2.0:1789760752.227457:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484261760 00000100:00100000:2.0:1789760752.227548:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741492788898 time 741520681385 state 0x0 ref 1 00000100:00100000:2.0:1789760752.227566:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741520941801 time 741520681385 state 0x1 ref 1 00000100:00100000:2.0:1789760752.227634:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741520941801 time 741558564208 state 0x0 ref 1 00000100:00100000:2.0:1789760752.227648:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484261760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760752.227697:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22261760, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:2.0:1789760752.227705:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.227721:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:2.0:1789760752.227725:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760752.227731:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22261760->22265855] (req 22261760->22265855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.227745:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22261760->22265855] (req 22261760->22265855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760752.270638:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972753 is committed 00002000:00010000:0.0:1789760752.270749:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22261760 -> 22265856 00010000:00010000:0.0:1789760752.270765:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22261760->22265855] (req 22261760->22265855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.270778:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22261760->22265855] (req 22261760->22265855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.270787:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22261760->22265855] (req 22261760->22265855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.270794:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.270801:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22261760->22265855] (req 22261760->22265855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.270810:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22261760->22265855] (req 22261760->22265855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.270831:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22261760->22265855] (req 22261760->22265855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.270838:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208e5a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22261760->22265855] (req 22261760->22265855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.270932:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484261760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43282us (43655us total) trans 4294972753 rc 0/0 00000100:00100000:0.0:1789760752.270942:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5464 00000100:00100000:0.0:1789760752.270952:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741520941801 time 741558564208 state 0x2 ref 0 00000100:00100000:1.0:1789760752.275040:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760752.275252:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484261888 00000100:00100000:2.0:1789760752.275261:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484261888 00000100:00100000:2.0:1789760752.275317:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741520941801 time 741558564208 state 0x0 ref 1 00000100:00100000:2.0:1789760752.275332:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741558824624 time 741558564208 state 0x1 ref 1 00000100:00100000:2.0:1789760752.275362:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741558824624 time 741606292693 state 0x0 ref 1 00000100:00100000:2.0:1789760752.275375:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484261888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760752.275421:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22265856, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:2.0:1789760752.275431:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.275446:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:2.0:1789760752.275451:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760752.275457:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22265856->22269951] (req 22265856->22269951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.275471:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22265856->22269951] (req 22265856->22269951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760752.346723:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972754 is committed 00002000:00010000:0.0:1789760752.346867:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22265856 -> 22269952 00010000:00010000:0.0:1789760752.346883:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22265856->22269951] (req 22265856->22269951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.346896:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22265856->22269951] (req 22265856->22269951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.346905:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22265856->22269951] (req 22265856->22269951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.346918:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.346923:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22265856->22269951] (req 22265856->22269951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.346932:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22265856->22269951] (req 22265856->22269951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.346951:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22265856->22269951] (req 22265856->22269951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.346958:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208e61 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22265856->22269951] (req 22265856->22269951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.347046:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484261888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 71669us (72022us total) trans 4294972754 rc 0/0 00000100:00100000:0.0:1789760752.347055:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5465 00000100:00100000:0.0:1789760752.347107:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741558824624 time 741606292693 state 0x2 ref 0 00000100:00100000:2.0:1789760752.351358:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.358464:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484262016 00000100:00100000:0.0:1789760752.358479:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484262016 00000100:00100000:0.0:1789760752.358544:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741558824624 time 741606292693 state 0x0 ref 1 00000100:00100000:0.0:1789760752.360678:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741606553109 time 741606292693 state 0x1 ref 1 00000100:00100000:0.0:1789760752.360730:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741606553109 time 741691660756 state 0x0 ref 1 00000100:00100000:0.0:1789760752.360761:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484262016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760752.360819:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22269952, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:0.0:1789760752.360828:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.360843:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:0.0:1789760752.360847:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760752.360852:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22269952->22274047] (req 22269952->22274047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.360865:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22269952->22274047] (req 22269952->22274047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760752.432765:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972755 is committed 00002000:00010000:0.0:1789760752.432878:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22269952 -> 22274048 00010000:00010000:0.0:1789760752.432893:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22269952->22274047] (req 22269952->22274047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.432907:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22269952->22274047] (req 22269952->22274047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.432917:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22269952->22274047] (req 22269952->22274047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.432924:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.432929:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22269952->22274047] (req 22269952->22274047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.432940:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22269952->22274047] (req 22269952->22274047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.432959:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22269952->22274047] (req 22269952->22274047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.432967:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e68 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22269952->22274047] (req 22269952->22274047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.433199:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484262016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 72450us (81845us total) trans 4294972755 rc 0/0 00000100:00100000:0.0:1789760752.433210:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5466 00000100:00100000:0.0:1789760752.433219:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741606553109 time 741691660756 state 0x2 ref 0 00000100:00100000:3.0:1789760752.443982:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.444427:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484262144 00000100:00100000:0.0:1789760752.444441:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484262144 00000100:00100000:0.0:1789760752.444517:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741606553109 time 741691660756 state 0x0 ref 1 00000100:00100000:0.0:1789760752.444536:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741691921172 time 741691660756 state 0x1 ref 1 00000100:00100000:0.0:1789760752.444580:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741691921172 time 741775510122 state 0x0 ref 1 00000100:00100000:0.0:1789760752.444620:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484262144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760752.444678:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22274048, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:0.0:1789760752.444686:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.444703:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:0.0:1789760752.444708:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760752.444713:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22274048->22278143] (req 22274048->22278143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.444727:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22274048->22278143] (req 22274048->22278143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760752.495753:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972756 is committed 00002000:00010000:3.0:1789760752.496013:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22274048 -> 22278144 00010000:00010000:3.0:1789760752.496037:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22274048->22278143] (req 22274048->22278143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.496048:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22274048->22278143] (req 22274048->22278143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.496057:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22274048->22278143] (req 22274048->22278143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.496088:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760752.496093:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22274048->22278143] (req 22274048->22278143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.496104:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22274048->22278143] (req 22274048->22278143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.496122:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22274048->22278143] (req 22274048->22278143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.496130:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208e6f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22274048->22278143] (req 22274048->22278143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760752.496222:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484262144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51604us (52242us total) trans 4294972756 rc 0/0 00000100:00100000:3.0:1789760752.496241:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5467 00000100:00100000:3.0:1789760752.496251:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741691921172 time 741775510122 state 0x2 ref 0 00000100:00100000:3.0:1789760752.501271:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760752.501336:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484262272 00000100:00100000:1.0:1789760752.501346:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484262272 00000100:00100000:1.0:1789760752.501400:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741691921172 time 741775510122 state 0x0 ref 1 00000100:00100000:1.0:1789760752.501419:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741775770538 time 741775510122 state 0x1 ref 1 00000100:00100000:1.0:1789760752.501454:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741775770538 time 741832384769 state 0x0 ref 1 00000100:00100000:1.0:1789760752.501466:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484262272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760752.501515:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22278144, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:1.0:1789760752.501523:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.501539:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:1.0:1789760752.501543:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760752.501548:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22278144->22282239] (req 22278144->22282239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.501561:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22278144->22282239] (req 22278144->22282239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.524264:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:2.0:1789760752.539895:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22278144 -> 22282240 00010000:00010000:2.0:1789760752.539914:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22278144->22282239] (req 22278144->22282239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.539927:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22278144->22282239] (req 22278144->22282239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.539936:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22278144->22282239] (req 22278144->22282239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.539943:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760752.539949:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22278144->22282239] (req 22278144->22282239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.539957:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22278144->22282239] (req 22278144->22282239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.539976:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22278144->22282239] (req 22278144->22282239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.539984:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208e76 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22278144->22282239] (req 22278144->22282239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760752.540111:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484262272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38641us (38844us total) trans 4294972757 rc 0/0 00000100:00100000:2.0:1789760752.540122:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5468 00000100:00100000:2.0:1789760752.540131:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741775770538 time 741832384769 state 0x2 ref 0 00000001:00080000:3.0:1789760752.540267:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972757 is committed 00000100:00100000:2.0:1789760752.543651:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.543723:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484262400 00000100:00100000:0.0:1789760752.543732:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484262400 00000100:00100000:0.0:1789760752.543787:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741775770538 time 741832384769 state 0x0 ref 1 00000100:00100000:0.0:1789760752.543803:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741832645185 time 741832384769 state 0x1 ref 1 00000100:00100000:0.0:1789760752.543832:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741832645185 time 741874762624 state 0x0 ref 1 00000100:00100000:0.0:1789760752.543845:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484262400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760752.543890:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22282240, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:0.0:1789760752.543899:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.543914:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:0.0:1789760752.543919:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760752.543925:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22282240->22286335] (req 22282240->22286335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.543938:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22282240->22286335] (req 22282240->22286335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760752.583358:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22282240 -> 22286336 00010000:00010000:0.0:1789760752.583378:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22282240->22286335] (req 22282240->22286335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.583392:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22282240->22286335] (req 22282240->22286335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.583401:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22282240->22286335] (req 22282240->22286335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.583408:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.583412:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22282240->22286335] (req 22282240->22286335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.583421:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22282240->22286335] (req 22282240->22286335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.583441:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22282240->22286335] (req 22282240->22286335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.583450:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396208e7d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22282240->22286335] (req 22282240->22286335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.583544:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484262400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39698us (39898us total) trans 4294972758 rc 0/0 00000001:00080000:2.0:1789760752.583548:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972758 is committed 00000100:00100000:0.0:1789760752.583554:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5469 00000100:00100000:0.0:1789760752.583563:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741832645185 time 741874762624 state 0x2 ref 0 00000100:00100000:0.0:1789760752.586782:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760752.586841:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484262528 00000100:00100000:2.0:1789760752.586852:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484262528 00000100:00100000:2.0:1789760752.586964:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741832645185 time 741874762624 state 0x0 ref 1 00000100:00100000:2.0:1789760752.586981:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741875023040 time 741874762624 state 0x1 ref 1 00000100:00100000:2.0:1789760752.587009:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741875023040 time 741917940034 state 0x0 ref 1 00000100:00100000:2.0:1789760752.587142:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484262528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760752.587188:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22286336, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:2.0:1789760752.587196:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.587212:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:2.0:1789760752.587216:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760752.587223:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22286336->22290431] (req 22286336->22290431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.587247:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22286336->22290431] (req 22286336->22290431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760752.627814:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972759 is committed 00002000:00010000:0.0:1789760752.628128:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22286336 -> 22290432 00010000:00010000:0.0:1789760752.628142:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22286336->22290431] (req 22286336->22290431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.628155:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22286336->22290431] (req 22286336->22290431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.628164:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22286336->22290431] (req 22286336->22290431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.628171:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.628175:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22286336->22290431] (req 22286336->22290431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.628185:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22286336->22290431] (req 22286336->22290431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.628204:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22286336->22290431] (req 22286336->22290431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.628344:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396208e84 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22286336->22290431] (req 22286336->22290431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.628438:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484262528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41295us (41657us total) trans 4294972759 rc 0/0 00000100:00100000:0.0:1789760752.628448:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5470 00000100:00100000:0.0:1789760752.628457:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741875023040 time 741917940034 state 0x2 ref 0 00000100:00100000:3.0:1789760752.631666:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760752.631877:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484262656 00000100:00100000:1.0:1789760752.631887:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484262656 00000100:00100000:1.0:1789760752.631938:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741875023040 time 741917940034 state 0x0 ref 1 00000100:00100000:1.0:1789760752.631954:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741918200450 time 741917940034 state 0x1 ref 1 00000100:00100000:1.0:1789760752.631984:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741918200450 time 741962914316 state 0x0 ref 1 00000100:00100000:1.0:1789760752.632000:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484262656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760752.632058:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22290432, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:1.0:1789760752.632440:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.632456:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=741s v=(0 0) 00010000:00010000:1.0:1789760752.632461:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760752.632468:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22290432->22294527] (req 22290432->22294527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760752.632482:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22290432->22294527] (req 22290432->22294527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760752.675376:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972760 is committed 00002000:00010000:2.0:1789760752.675477:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22290432 -> 22294528 00010000:00010000:2.0:1789760752.675492:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22290432->22294527] (req 22290432->22294527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.675505:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22290432->22294527] (req 22290432->22294527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.675520:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22290432->22294527] (req 22290432->22294527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.675534:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760752.675539:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22290432->22294527] (req 22290432->22294527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.675547:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22290432->22294527] (req 22290432->22294527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.675569:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22290432->22294527] (req 22290432->22294527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.675576:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396208e8b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22290432->22294527] (req 22290432->22294527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760752.675776:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484262656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43773us (44114us total) trans 4294972760 rc 0/0 00000100:00100000:2.0:1789760752.675786:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5471 00000100:00100000:2.0:1789760752.675796:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741918200450 time 741962914316 state 0x2 ref 0 00000100:00100000:2.0:1789760752.680565:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.680647:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484262784 00000100:00100000:0.0:1789760752.680657:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484262784 00000100:00100000:0.0:1789760752.680714:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741918200450 time 741962914316 state 0x0 ref 1 00000100:00100000:0.0:1789760752.680730:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741963174732 time 741962914316 state 0x1 ref 1 00000100:00100000:0.0:1789760752.680759:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741963174732 time 742011690388 state 0x0 ref 1 00000100:00100000:0.0:1789760752.680773:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484262784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760752.680819:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22294528, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:0.0:1789760752.680828:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.680845:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:0.0:1789760752.680853:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760752.680861:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22294528->22298623] (req 22294528->22298623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.680875:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22294528->22298623] (req 22294528->22298623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760752.712880:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22294528 -> 22298624 00010000:00010000:0.0:1789760752.712899:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22294528->22298623] (req 22294528->22298623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.712911:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22294528->22298623] (req 22294528->22298623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.712920:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22294528->22298623] (req 22294528->22298623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.712928:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.712933:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22294528->22298623] (req 22294528->22298623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.712943:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22294528->22298623] (req 22294528->22298623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.712963:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22294528->22298623] (req 22294528->22298623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.712971:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208e92 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22294528->22298623] (req 22294528->22298623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.713166:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484262784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32391us (32605us total) trans 4294972761 rc 0/0 00000100:00100000:0.0:1789760752.713176:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5472 00000100:00100000:0.0:1789760752.713186:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741963174732 time 742011690388 state 0x2 ref 0 00000001:00080000:0.0:1789760752.713227:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972761 is committed 00000100:00100000:0.0:1789760752.718249:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760752.718361:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484262912 00000100:00100000:2.0:1789760752.718371:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484262912 00000100:00100000:2.0:1789760752.718425:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 741963174732 time 742011690388 state 0x0 ref 1 00000100:00100000:2.0:1789760752.718439:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742011950804 time 742011690388 state 0x1 ref 1 00000100:00100000:2.0:1789760752.718471:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742011950804 time 742049401734 state 0x0 ref 1 00000100:00100000:2.0:1789760752.718484:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484262912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760752.718536:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22298624, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:2.0:1789760752.718545:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.718565:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:2.0:1789760752.718570:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760752.718576:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22298624->22302719] (req 22298624->22302719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.718589:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22298624->22302719] (req 22298624->22302719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760752.781055:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972762 is committed 00002000:00010000:3.0:1789760752.781115:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22298624 -> 22302720 00010000:00010000:3.0:1789760752.781130:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22298624->22302719] (req 22298624->22302719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.781143:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22298624->22302719] (req 22298624->22302719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.781152:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22298624->22302719] (req 22298624->22302719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.781160:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760752.781165:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22298624->22302719] (req 22298624->22302719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.781175:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22298624->22302719] (req 22298624->22302719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.781194:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22298624->22302719] (req 22298624->22302719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.781201:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396208e99 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22298624->22302719] (req 22298624->22302719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760752.781300:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484262912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62814us (63066us total) trans 4294972762 rc 0/0 00000100:00100000:3.0:1789760752.781310:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5473 00000100:00100000:3.0:1789760752.781327:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742011950804 time 742049401734 state 0x2 ref 0 00000100:00100000:3.0:1789760752.787908:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760752.787999:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484263040 00000100:00100000:2.0:1789760752.788008:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484263040 00000100:00100000:2.0:1789760752.788061:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742011950804 time 742049401734 state 0x0 ref 1 00000100:00100000:2.0:1789760752.788103:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742049662150 time 742049401734 state 0x1 ref 1 00000100:00100000:2.0:1789760752.788127:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742049662150 time 742119058048 state 0x0 ref 1 00000100:00100000:2.0:1789760752.788139:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484263040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760752.788184:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22302720, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:2.0:1789760752.788191:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.788203:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:2.0:1789760752.788206:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760752.788211:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22302720->22306815] (req 22302720->22306815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.788228:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22302720->22306815] (req 22302720->22306815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760752.817428:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22302720 -> 22306816 00010000:00010000:2.0:1789760752.817447:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22302720->22306815] (req 22302720->22306815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.817460:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22302720->22306815] (req 22302720->22306815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.817468:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22302720->22306815] (req 22302720->22306815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.817475:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760752.817480:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22302720->22306815] (req 22302720->22306815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.817488:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22302720->22306815] (req 22302720->22306815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.817506:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22302720->22306815] (req 22302720->22306815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.817513:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396208ea0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22302720->22306815] (req 22302720->22306815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760752.817599:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484263040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29458us (29696us total) trans 4294972763 rc 0/0 00000100:00100000:2.0:1789760752.817608:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5474 00000100:00100000:2.0:1789760752.817617:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742049662150 time 742119058048 state 0x2 ref 0 00000001:00080000:2.0:1789760752.817654:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972763 is committed 00000100:00100000:0.0:1789760752.824960:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760752.825122:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484263168 00000100:00100000:3.0:1789760752.825132:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484263168 00000100:00100000:3.0:1789760752.825185:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742049662150 time 742119058048 state 0x0 ref 1 00000100:00100000:3.0:1789760752.825199:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742119318464 time 742119058048 state 0x1 ref 1 00000100:00100000:3.0:1789760752.825238:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742119318464 time 742156161282 state 0x0 ref 1 00000100:00100000:3.0:1789760752.825250:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484263168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760752.825293:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22306816, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:3.0:1789760752.825301:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.825316:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:3.0:1789760752.825320:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760752.825325:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22306816->22310911] (req 22306816->22310911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.825339:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22306816->22310911] (req 22306816->22310911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760752.859614:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22306816 -> 22310912 00000001:00080000:1.0:1789760752.859618:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972764 is committed 00010000:00010000:0.0:1789760752.859631:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22306816->22310911] (req 22306816->22310911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.859644:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22306816->22310911] (req 22306816->22310911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.859653:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22306816->22310911] (req 22306816->22310911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.859661:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760752.859665:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22306816->22310911] (req 22306816->22310911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.859736:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22306816->22310911] (req 22306816->22310911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.859763:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22306816->22310911] (req 22306816->22310911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.859771:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396208ea7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22306816->22310911] (req 22306816->22310911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.859862:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484263168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34608us (34904us total) trans 4294972764 rc 0/0 00000100:00100000:0.0:1789760752.859872:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5475 00000100:00100000:0.0:1789760752.859880:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742119318464 time 742156161282 state 0x2 ref 0 00000100:00100000:2.0:1789760752.863396:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.864150:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484263296 00000100:00100000:0.0:1789760752.864160:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484263296 00000100:00100000:0.0:1789760752.864220:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742119318464 time 742156161282 state 0x0 ref 1 00000100:00100000:0.0:1789760752.864245:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742156421698 time 742156161282 state 0x1 ref 1 00000100:00100000:0.0:1789760752.864276:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742156421698 time 742195207302 state 0x0 ref 1 00000100:00100000:0.0:1789760752.864291:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484263296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760752.864338:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22310912, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:0.0:1789760752.864347:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.864363:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:0.0:1789760752.864367:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760752.864374:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22310912->22315007] (req 22310912->22315007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.864387:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22310912->22315007] (req 22310912->22315007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760752.870284:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760752.870358:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484263424 00000100:00100000:2.0:1789760752.870368:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484263424 00000100:00100000:0.0:1789760752.870373:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.870393:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.870409:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760752.870412:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 483 00000100:00100000:2.0:1789760752.870421:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b7050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484263424:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760752.870427:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.870445:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484263680 00000100:00100000:0.0:1789760752.870448:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484263680 00000100:00100000:0.0:1789760752.870483:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484263936 00000100:00100000:2.0:1789760752.870484:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b7050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484263424:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 64us (207us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.870486:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484263936 00000100:00100000:2.0:1789760752.870491:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 483 00000100:00100000:0.0:1789760752.870502:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484263552 00000100:00100000:0.0:1789760752.870504:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484263552 00000100:00100000:0.0:1789760752.870517:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484263808 00000100:00100000:0.0:1789760752.870519:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484263808 00000100:00100000:2.0:1789760752.870535:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 941 00000100:00100000:2.0:1789760752.870542:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484263680:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760752.870544:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 942 00000100:00100000:0.0:1789760752.870550:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484263936:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760752.870572:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484263936:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 22us (178us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760752.870572:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484263680:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 30us (198us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.870578:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 942 00000100:00100000:2.0:1789760752.870581:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 941 00000100:00100000:0.0:1789760752.870590:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 943 00000100:00100000:0.0:1789760752.870597:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484263552:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760752.870620:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484263552:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 25us (211us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.870627:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 943 00000100:00100000:0.0:1789760752.870638:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 944 00000100:00100000:0.0:1789760752.870645:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484263808:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760752.870678:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5080700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484263808:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 31us (246us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760752.870684:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 944 00000100:00080000:3.0:1789760752.871348:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:3.0:1789760752.871358:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000001:00080000:2.0:1789760752.909392:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972765 is committed 00002000:00010000:3.0:1789760752.909483:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22310912 -> 22315008 00010000:00010000:3.0:1789760752.909499:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22310912->22315007] (req 22310912->22315007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.909512:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22310912->22315007] (req 22310912->22315007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.909521:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22310912->22315007] (req 22310912->22315007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.909528:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760752.909534:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22310912->22315007] (req 22310912->22315007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.909544:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22310912->22315007] (req 22310912->22315007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.909563:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22310912->22315007] (req 22310912->22315007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760752.909571:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396208eae lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22310912->22315007] (req 22310912->22315007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760752.909819:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484263296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45527us (46426us total) trans 4294972765 rc 0/0 00000100:00100000:3.0:1789760752.909829:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5476 00000100:00100000:3.0:1789760752.909839:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742156421698 time 742195207302 state 0x2 ref 0 00000100:00100000:0.0:1789760752.913575:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760752.914579:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484264064 00000100:00100000:2.0:1789760752.914590:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484264064 00000100:00100000:2.0:1789760752.914659:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742156421698 time 742195207302 state 0x0 ref 1 00000100:00100000:2.0:1789760752.914675:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742195467718 time 742195207302 state 0x1 ref 1 00000100:00100000:2.0:1789760752.914709:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742195467718 time 742245639998 state 0x0 ref 1 00000100:00100000:2.0:1789760752.914723:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484264064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760752.914772:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22315008, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:2.0:1789760752.914781:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.914795:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:2.0:1789760752.914801:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760752.914807:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22315008->22319103] (req 22315008->22319103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.914821:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22315008->22319103] (req 22315008->22319103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760752.945474:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972766 is committed 00002000:00010000:2.0:1789760752.945802:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22315008 -> 22319104 00010000:00010000:2.0:1789760752.945818:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22315008->22319103] (req 22315008->22319103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.945831:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22315008->22319103] (req 22315008->22319103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.945840:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22315008->22319103] (req 22315008->22319103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.945848:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760752.945853:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22315008->22319103] (req 22315008->22319103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.945863:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22315008->22319103] (req 22315008->22319103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.945881:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22315008->22319103] (req 22315008->22319103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760752.945889:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396208eb5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22315008->22319103] (req 22315008->22319103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760752.946024:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484264064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31291us (32446us total) trans 4294972766 rc 0/0 00000100:00100000:2.0:1789760752.946035:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5477 00000100:00100000:2.0:1789760752.946046:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742195467718 time 742245639998 state 0x2 ref 0 00000100:00100000:2.0:1789760752.949618:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760752.949709:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484264192 00000100:00100000:0.0:1789760752.949719:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484264192 00000100:00100000:0.0:1789760752.949778:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742195467718 time 742245639998 state 0x0 ref 1 00000100:00100000:0.0:1789760752.949792:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742245900414 time 742245639998 state 0x1 ref 1 00000100:00100000:0.0:1789760752.949852:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742245900414 time 742280782585 state 0x0 ref 1 00000100:00100000:0.0:1789760752.949866:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484264192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760752.949912:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22319104, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:0.0:1789760752.949920:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.949935:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:0.0:1789760752.949940:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760752.949947:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22319104->22323199] (req 22319104->22323199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760752.949966:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22319104->22323199] (req 22319104->22323199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760753.010056:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22319104 -> 22323200 00010000:00010000:0.0:1789760753.010097:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22319104->22323199] (req 22319104->22323199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.010109:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22319104->22323199] (req 22319104->22323199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.010118:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22319104->22323199] (req 22319104->22323199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.010125:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760753.010129:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22319104->22323199] (req 22319104->22323199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.010138:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22319104->22323199] (req 22319104->22323199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.010158:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22319104->22323199] (req 22319104->22323199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.010166:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396208ebc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22319104->22323199] (req 22319104->22323199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760753.010263:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484264192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60395us (60649us total) trans 4294972767 rc 0/0 00000100:00100000:0.0:1789760753.010273:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5478 00000100:00100000:0.0:1789760753.010283:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742245900414 time 742280782585 state 0x2 ref 0 00000001:00080000:1.0:1789760753.011315:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972767 is committed 00000100:00100000:0.0:1789760753.013960:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760753.014041:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484264320 00000100:00100000:3.0:1789760753.014049:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484264320 00000100:00100000:3.0:1789760753.014131:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742245900414 time 742280782585 state 0x0 ref 1 00000100:00100000:3.0:1789760753.014144:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742281043001 time 742280782585 state 0x1 ref 1 00000100:00100000:3.0:1789760753.014270:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742281043001 time 742345200637 state 0x0 ref 1 00000100:00100000:3.0:1789760753.014283:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484264320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760753.014336:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22323200, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:3.0:1789760753.014345:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.014360:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:3.0:1789760753.014364:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760753.014370:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22323200->22327295] (req 22323200->22327295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.014383:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22323200->22327295] (req 22323200->22327295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760753.058120:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972768 is committed 00002000:00010000:3.0:1789760753.058295:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22323200 -> 22327296 00010000:00010000:3.0:1789760753.058309:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22323200->22327295] (req 22323200->22327295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.058332:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22323200->22327295] (req 22323200->22327295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.058341:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22323200->22327295] (req 22323200->22327295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.058349:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760753.058353:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22323200->22327295] (req 22323200->22327295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.058362:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22323200->22327295] (req 22323200->22327295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.058386:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22323200->22327295] (req 22323200->22327295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.058394:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208ec3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22323200->22327295] (req 22323200->22327295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760753.058485:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484264320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44199us (44529us total) trans 4294972768 rc 0/0 00000100:00100000:3.0:1789760753.058496:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5479 00000100:00100000:3.0:1789760753.058506:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742281043001 time 742345200637 state 0x2 ref 0 00000100:00100000:1.0:1789760753.062768:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760753.062836:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484264448 00000100:00100000:3.0:1789760753.062845:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484264448 00000100:00100000:3.0:1789760753.062896:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742281043001 time 742345200637 state 0x0 ref 1 00000100:00100000:3.0:1789760753.062909:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742345461053 time 742345200637 state 0x1 ref 1 00000100:00100000:3.0:1789760753.062937:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742345461053 time 742393868123 state 0x0 ref 1 00000100:00100000:3.0:1789760753.062949:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484264448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760753.062997:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22327296, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:3.0:1789760753.063006:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.063029:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:3.0:1789760753.063035:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760753.063041:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22327296->22331391] (req 22327296->22331391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.063056:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22327296->22331391] (req 22327296->22331391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760753.138040:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22327296 -> 22331392 00010000:00010000:0.0:1789760753.138059:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22327296->22331391] (req 22327296->22331391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.138110:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22327296->22331391] (req 22327296->22331391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.138120:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22327296->22331391] (req 22327296->22331391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.138127:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760753.138132:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22327296->22331391] (req 22327296->22331391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.138142:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22327296->22331391] (req 22327296->22331391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.138162:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22327296->22331391] (req 22327296->22331391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.138174:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208eca lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22327296->22331391] (req 22327296->22331391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760753.138179:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972769 is committed 00000100:00100000:0.0:1789760753.138266:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484264448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 75314us (75504us total) trans 4294972769 rc 0/0 00000100:00100000:0.0:1789760753.138276:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5480 00000100:00100000:0.0:1789760753.138286:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742345461053 time 742393868123 state 0x2 ref 0 00000100:00100000:0.0:1789760753.142497:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760753.146181:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484264576 00000100:00100000:2.0:1789760753.146196:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484264576 00000100:00100000:2.0:1789760753.146267:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742345461053 time 742393868123 state 0x0 ref 1 00000100:00100000:2.0:1789760753.146284:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742394128539 time 742393868123 state 0x1 ref 1 00000100:00100000:2.0:1789760753.146390:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742394128539 time 742477319623 state 0x0 ref 1 00000100:00100000:2.0:1789760753.146404:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484264576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760753.147141:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22331392, mtime: 0x6aad94f0, atime: 0x0, ctime: 0x6aad94f0, blocks: 0xb007 00010000:00010000:2.0:1789760753.147154:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.147169:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:2.0:1789760753.147174:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760753.147181:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22331392->22335487] (req 22331392->22335487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.147196:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22331392->22335487] (req 22331392->22335487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760753.193806:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972770 is committed 00002000:00010000:0.0:1789760753.193903:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22331392 -> 22335488 00002000:00010000:0.0:1789760753.193908:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760752 -> 1789760753 00002000:00010000:0.0:1789760753.193910:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760752 -> 1789760753 00010000:00010000:0.0:1789760753.193923:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22331392->22335487] (req 22331392->22335487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.193935:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22331392->22335487] (req 22331392->22335487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.193944:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22331392->22335487] (req 22331392->22335487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.193951:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760753.193956:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22331392->22335487] (req 22331392->22335487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.193969:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22331392->22335487] (req 22331392->22335487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.193989:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22331392->22335487] (req 22331392->22335487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.193997:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396208ed1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22331392->22335487] (req 22331392->22335487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760753.194267:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484264576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47860us (51841us total) trans 4294972770 rc 0/0 00000100:00100000:0.0:1789760753.194278:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5481 00000100:00100000:0.0:1789760753.194287:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742394128539 time 742477319623 state 0x2 ref 0 00000100:00100000:2.0:1789760753.200692:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760753.202260:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484264704 00000100:00100000:0.0:1789760753.202273:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484264704 00000100:00100000:0.0:1789760753.202339:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742394128539 time 742477319623 state 0x0 ref 1 00000100:00100000:0.0:1789760753.202354:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742477580039 time 742477319623 state 0x1 ref 1 00000100:00100000:0.0:1789760753.202390:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742477580039 time 742533320678 state 0x0 ref 1 00000100:00100000:0.0:1789760753.202404:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484264704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760753.202520:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22335488, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:0.0:1789760753.202530:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.202546:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:0.0:1789760753.202551:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760753.202557:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22335488->22339583] (req 22335488->22339583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.202582:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22335488->22339583] (req 22335488->22339583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760753.233874:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972771 is committed 00002000:00010000:2.0:1789760753.234022:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22335488 -> 22339584 00010000:00010000:2.0:1789760753.234035:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22335488->22339583] (req 22335488->22339583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.234044:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22335488->22339583] (req 22335488->22339583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.234050:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22335488->22339583] (req 22335488->22339583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.234055:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760753.234059:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22335488->22339583] (req 22335488->22339583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.234083:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22335488->22339583] (req 22335488->22339583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.234098:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22335488->22339583] (req 22335488->22339583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.234103:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208ed8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22335488->22339583] (req 22335488->22339583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760753.234188:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484264704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31784us (33502us total) trans 4294972771 rc 0/0 00000100:00100000:2.0:1789760753.234198:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5482 00000100:00100000:2.0:1789760753.234208:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742477580039 time 742533320678 state 0x2 ref 0 00000100:00100000:1.0:1789760753.242098:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760753.242174:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484264832 00000100:00100000:3.0:1789760753.242183:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484264832 00000100:00100000:3.0:1789760753.242241:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742477580039 time 742533320678 state 0x0 ref 1 00000100:00100000:3.0:1789760753.242255:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742533581094 time 742533320678 state 0x1 ref 1 00000100:00100000:3.0:1789760753.242283:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742533581094 time 742573214017 state 0x0 ref 1 00000100:00100000:3.0:1789760753.242295:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484264832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760753.242343:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22339584, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:3.0:1789760753.242352:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.242368:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:3.0:1789760753.242373:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760753.242380:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22339584->22343679] (req 22339584->22343679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.242395:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22339584->22343679] (req 22339584->22343679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760753.271060:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972772 is committed 00002000:00010000:2.0:1789760753.271266:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22339584 -> 22343680 00010000:00010000:2.0:1789760753.271281:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22339584->22343679] (req 22339584->22343679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.271294:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22339584->22343679] (req 22339584->22343679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.271303:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22339584->22343679] (req 22339584->22343679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.271311:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760753.271316:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22339584->22343679] (req 22339584->22343679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.271325:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22339584->22343679] (req 22339584->22343679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.271344:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22339584->22343679] (req 22339584->22343679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.271352:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208edf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22339584->22343679] (req 22339584->22343679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760753.271439:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484264832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29142us (29380us total) trans 4294972772 rc 0/0 00000100:00100000:2.0:1789760753.271450:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5483 00000100:00100000:2.0:1789760753.271460:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742533581094 time 742573214017 state 0x2 ref 0 00000100:00100000:0.0:1789760753.274951:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760753.275167:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484264960 00000100:00100000:3.0:1789760753.275184:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484264960 00000100:00100000:3.0:1789760753.275249:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742533581094 time 742573214017 state 0x0 ref 1 00000100:00100000:3.0:1789760753.275425:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742573474433 time 742573214017 state 0x1 ref 1 00000100:00100000:3.0:1789760753.275494:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742573474433 time 742606422897 state 0x0 ref 1 00000100:00100000:3.0:1789760753.275508:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484264960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760753.275560:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22343680, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:3.0:1789760753.275569:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.275593:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:3.0:1789760753.275598:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760753.275604:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22343680->22347775] (req 22343680->22347775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.275618:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22343680->22347775] (req 22343680->22347775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760753.317992:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972773 is committed 00002000:00010000:2.0:1789760753.321738:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22343680 -> 22347776 00010000:00010000:2.0:1789760753.321756:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22343680->22347775] (req 22343680->22347775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.321768:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22343680->22347775] (req 22343680->22347775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.321776:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22343680->22347775] (req 22343680->22347775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.321788:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760753.321792:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22343680->22347775] (req 22343680->22347775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.321801:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22343680->22347775] (req 22343680->22347775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.321821:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22343680->22347775] (req 22343680->22347775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.321830:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208ee6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22343680->22347775] (req 22343680->22347775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760753.321938:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484264960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46428us (46991us total) trans 4294972773 rc 0/0 00000100:00100000:2.0:1789760753.321950:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5484 00000100:00100000:2.0:1789760753.321960:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742573474433 time 742606422897 state 0x2 ref 0 00000100:00100000:3.0:1789760753.329381:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760753.329493:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484265088 00000100:00100000:0.0:1789760753.329503:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484265088 00000100:00100000:0.0:1789760753.329563:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742573474433 time 742606422897 state 0x0 ref 1 00000100:00100000:0.0:1789760753.329578:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742606683313 time 742606422897 state 0x1 ref 1 00000100:00100000:0.0:1789760753.329609:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742606683313 time 742660540387 state 0x0 ref 1 00000100:00100000:0.0:1789760753.329623:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484265088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760753.329671:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22347776, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:0.0:1789760753.329679:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.329694:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:0.0:1789760753.329699:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760753.329705:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22347776->22351871] (req 22347776->22351871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.329720:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22347776->22351871] (req 22347776->22351871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760753.370018:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22347776 -> 22351872 00010000:00010000:0.0:1789760753.370037:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22347776->22351871] (req 22347776->22351871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.370052:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22347776->22351871] (req 22347776->22351871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.370062:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22347776->22351871] (req 22347776->22351871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.370095:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760753.370100:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22347776->22351871] (req 22347776->22351871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.370110:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22347776->22351871] (req 22347776->22351871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.370128:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22347776->22351871] (req 22347776->22351871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.370137:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208eed lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22347776->22351871] (req 22347776->22351871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760753.370222:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484265088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40599us (40845us total) trans 4294972774 rc 0/0 00000100:00100000:0.0:1789760753.370244:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5485 00000100:00100000:0.0:1789760753.370254:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742606683313 time 742660540387 state 0x2 ref 0 00000001:00080000:2.0:1789760753.370644:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972774 is committed 00000100:00100000:1.0:1789760753.377861:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760753.377973:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484265216 00000100:00100000:2.0:1789760753.377982:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484265216 00000100:00100000:2.0:1789760753.378031:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742606683313 time 742660540387 state 0x0 ref 1 00000100:00100000:2.0:1789760753.378046:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742660800803 time 742660540387 state 0x1 ref 1 00000100:00100000:2.0:1789760753.378182:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742660800803 time 742709111931 state 0x0 ref 1 00000100:00100000:2.0:1789760753.378196:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484265216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760753.378255:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22351872, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:2.0:1789760753.378264:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.378280:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:2.0:1789760753.378285:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760753.378291:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22351872->22355967] (req 22351872->22355967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.378305:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22351872->22355967] (req 22351872->22355967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760753.431928:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972775 is committed 00002000:00010000:2.0:1789760753.432119:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22351872 -> 22355968 00010000:00010000:2.0:1789760753.432136:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22351872->22355967] (req 22351872->22355967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.432149:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22351872->22355967] (req 22351872->22355967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.432158:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22351872->22355967] (req 22351872->22355967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.432164:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760753.432169:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22351872->22355967] (req 22351872->22355967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.432252:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22351872->22355967] (req 22351872->22355967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.432272:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22351872->22355967] (req 22351872->22355967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.432279:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208ef4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22351872->22355967] (req 22351872->22355967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760753.432490:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484265216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54290us (54631us total) trans 4294972775 rc 0/0 00000100:00100000:2.0:1789760753.432504:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5486 00000100:00100000:2.0:1789760753.432516:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742660800803 time 742709111931 state 0x2 ref 0 00000100:00100000:1.0:1789760753.442373:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760753.444039:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484265344 00000100:00100000:2.0:1789760753.444147:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484265344 00000100:00100000:3.0:1789760753.444321:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742660800803 time 742709111931 state 0x0 ref 1 00000100:00100000:3.0:1789760753.444336:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742709372347 time 742709111931 state 0x1 ref 1 00000100:00100000:3.0:1789760753.444369:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742709372347 time 742775300235 state 0x0 ref 1 00000100:00100000:3.0:1789760753.444381:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484265344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760753.444546:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22355968, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:3.0:1789760753.444556:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.444688:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:3.0:1789760753.444694:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760753.444701:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22355968->22360063] (req 22355968->22360063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.444717:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22355968->22360063] (req 22355968->22360063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760753.493587:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972776 is committed 00002000:00010000:2.0:1789760753.493791:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22355968 -> 22360064 00010000:00010000:2.0:1789760753.493808:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22355968->22360063] (req 22355968->22360063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.493821:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22355968->22360063] (req 22355968->22360063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.493830:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22355968->22360063] (req 22355968->22360063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.493844:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760753.493849:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22355968->22360063] (req 22355968->22360063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.493857:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22355968->22360063] (req 22355968->22360063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.493877:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22355968->22360063] (req 22355968->22360063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.493884:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396208efb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22355968->22360063] (req 22355968->22360063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760753.493972:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484265344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49588us (51602us total) trans 4294972776 rc 0/0 00000100:00100000:2.0:1789760753.493982:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5487 00000100:00100000:2.0:1789760753.493993:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742709372347 time 742775300235 state 0x2 ref 0 00000100:00100000:3.0:1789760753.501365:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760753.503378:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484265472 00000100:00100000:2.0:1789760753.503391:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484265472 00000100:00100000:3.0:1789760753.504391:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742709372347 time 742775300235 state 0x0 ref 1 00000100:00100000:3.0:1789760753.504408:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742775560651 time 742775300235 state 0x1 ref 1 00000100:00100000:3.0:1789760753.504439:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742775560651 time 742835370556 state 0x0 ref 1 00000100:00100000:3.0:1789760753.504455:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484265472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760753.504504:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22360064, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:3.0:1789760753.504514:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.504735:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:3.0:1789760753.504741:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760753.504748:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22360064->22364159] (req 22360064->22364159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.504764:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22360064->22364159] (req 22360064->22364159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760753.543804:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22360064 -> 22364160 00010000:00010000:0.0:1789760753.543823:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22360064->22364159] (req 22360064->22364159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.543837:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22360064->22364159] (req 22360064->22364159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.543847:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22360064->22364159] (req 22360064->22364159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.543854:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00000001:00080000:1.0:1789760753.543857:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972777 is committed 00010000:00010000:0.0:1789760753.543859:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22360064->22364159] (req 22360064->22364159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.543867:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22360064->22364159] (req 22360064->22364159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.543887:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22360064->22364159] (req 22360064->22364159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.543901:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396208f02 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22360064->22364159] (req 22360064->22364159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760753.543992:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484265472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39534us (42631us total) trans 4294972777 rc 0/0 00000100:00100000:0.0:1789760753.544002:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5488 00000100:00100000:0.0:1789760753.544012:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742775560651 time 742835370556 state 0x2 ref 0 00000100:00100000:0.0:1789760753.548954:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508926592:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760753.549032:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760753.549204:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508926592 00000100:00100000:1.0:1789760753.549212:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508926592 00000100:00100000:1.0:1789760753.549269:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 390 00000100:00100000:1.0:1789760753.549279:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508926592:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:1.0:1789760753.549302:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 526450688 pending 0 free 3835691008 avail 3794796544 00000100:00100000:1.0:1789760753.549344:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508926592:12345-0@lo:13:osp-pre-0-1.0 Request processed in 66us (312us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760753.549351:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 390 00000100:00100000:3.0:1789760753.549366:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508926720:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760753.549395:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760753.549432:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508926720 00000100:00100000:1.0:1789760753.549436:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508926720 00000100:00100000:1.0:1789760753.549456:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 391 00010000:00010000:3.0:1789760753.549458:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760753.549462:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508926720:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:1.0:1789760753.549472:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 526450688 pending 0 free 3835691008 avail 3794796544 00000100:00100000:1.0:1789760753.549506:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508926720:12345-0@lo:13:osp-pre-0-0.0 Request processed in 43us (109us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760753.549512:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 391 00000100:00100000:0.0:1789760753.550138:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508926592:0@lo:13:osp-pre-0-1.0 00000100:00100000:3.0:1789760753.550395:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e87100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508926848:0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760753.550440:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760753.550457:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760753.550504:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508926720:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760753.550611:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508926848 00000100:00100000:0.0:1789760753.550617:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508926848 00000100:00100000:2.0:1789760753.550661:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484265600 00000100:00100000:2.0:1789760753.550667:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484265600 00000100:00100000:0.0:1789760753.550668:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 449 00000100:00100000:0.0:1789760753.550681:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508926848:12345-0@lo:41:osp-pre-0-1.0 00000100:00100000:2.0:1789760753.550710:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742775560651 time 742835370556 state 0x0 ref 1 00000004:00000024:0.0:1789760753.550716:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:2.0:1789760753.550726:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742835630972 time 742835370556 state 0x1 ref 1 00000100:00100000:2.0:1789760753.550752:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742835630972 time 742881682966 state 0x0 ref 1 00000100:00100000:0.0:1789760753.550761:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508926848:12345-0@lo:41:osp-pre-0-1.0 Request processed in 82us (321us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760753.550767:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484265600:12345-192.168.203.10@tcp:4:dd.0 00000100:00100000:0.0:1789760753.550774:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 449 00002000:00010000:2.0:1789760753.550816:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22364160, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:2.0:1789760753.550826:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760753.550837:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e87100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508926848:0@lo:41:osp-pre-0-1.0 00010000:00010000:2.0:1789760753.550844:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:2.0:1789760753.550849:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760753.550856:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22364160->22368255] (req 22364160->22368255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.550872:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22364160->22368255] (req 22364160->22368255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760753.595187:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972778 is committed 00002000:00010000:2.0:1789760753.595277:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22364160 -> 22368256 00010000:00010000:2.0:1789760753.595293:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22364160->22368255] (req 22364160->22368255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.595306:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22364160->22368255] (req 22364160->22368255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.595315:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22364160->22368255] (req 22364160->22368255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.595323:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760753.595328:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22364160->22368255] (req 22364160->22368255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.595337:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22364160->22368255] (req 22364160->22368255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.595357:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22364160->22368255] (req 22364160->22368255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.595365:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f09 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22364160->22368255] (req 22364160->22368255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760753.595491:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484265600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44724us (45030us total) trans 4294972778 rc 0/0 00000100:00100000:2.0:1789760753.595501:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5489 00000100:00100000:2.0:1789760753.595510:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742835630972 time 742881682966 state 0x2 ref 0 00000100:00100000:1.0:1789760753.601239:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760753.601324:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484265728 00000100:00100000:0.0:1789760753.601333:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484265728 00000100:00100000:0.0:1789760753.601395:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742835630972 time 742881682966 state 0x0 ref 1 00000100:00100000:0.0:1789760753.601410:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742881943382 time 742881682966 state 0x1 ref 1 00000100:00100000:0.0:1789760753.601447:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742881943382 time 742932377489 state 0x0 ref 1 00000100:00100000:0.0:1789760753.601460:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484265728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760753.601510:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22368256, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:0.0:1789760753.601519:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.601535:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:0.0:1789760753.601540:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760753.601547:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22368256->22372351] (req 22368256->22372351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.601561:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22368256->22372351] (req 22368256->22372351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760753.634337:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22368256 -> 22372352 00010000:00010000:1.0:1789760753.634368:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22368256->22372351] (req 22368256->22372351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.634381:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22368256->22372351] (req 22368256->22372351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.634391:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22368256->22372351] (req 22368256->22372351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.634408:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760753.634413:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22368256->22372351] (req 22368256->22372351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.634422:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22368256->22372351] (req 22368256->22372351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.634442:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22368256->22372351] (req 22368256->22372351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.634450:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396208f10 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22368256->22372351] (req 22368256->22372351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760753.634535:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484265728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33073us (33311us total) trans 4294972779 rc 0/0 00000100:00100000:1.0:1789760753.634546:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5490 00000100:00100000:1.0:1789760753.634556:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742881943382 time 742932377489 state 0x2 ref 0 00000001:00080000:3.0:1789760753.638258:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972779 is committed 00000100:00100000:1.0:1789760753.639185:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760753.639289:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484265856 00000100:00100000:2.0:1789760753.639300:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484265856 00000100:00100000:2.0:1789760753.639355:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742881943382 time 742932377489 state 0x0 ref 1 00000100:00100000:2.0:1789760753.639378:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742932637905 time 742932377489 state 0x1 ref 1 00000100:00100000:2.0:1789760753.639403:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742932637905 time 742970334066 state 0x0 ref 1 00000100:00100000:2.0:1789760753.639416:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484265856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760753.639467:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22372352, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:2.0:1789760753.639476:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.639492:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=742s v=(0 0) 00010000:00010000:2.0:1789760753.639497:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760753.639504:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22372352->22376447] (req 22372352->22376447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.639519:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22372352->22376447] (req 22372352->22376447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760753.671420:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972780 is committed 00002000:00010000:2.0:1789760753.671523:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22372352 -> 22376448 00010000:00010000:2.0:1789760753.671540:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22372352->22376447] (req 22372352->22376447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.671553:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22372352->22376447] (req 22372352->22376447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.671562:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22372352->22376447] (req 22372352->22376447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.671570:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760753.671575:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22372352->22376447] (req 22372352->22376447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.671586:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22372352->22376447] (req 22372352->22376447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.671680:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22372352->22376447] (req 22372352->22376447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.671688:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f17 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22372352->22376447] (req 22372352->22376447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760753.671978:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484265856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32560us (32796us total) trans 4294972780 rc 0/0 00000100:00100000:2.0:1789760753.671989:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5491 00000100:00100000:2.0:1789760753.671998:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742932637905 time 742970334066 state 0x2 ref 0 00000100:00100000:1.0:1789760753.675923:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760753.676303:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484265984 00000100:00100000:3.0:1789760753.676313:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484265984 00000100:00100000:3.0:1789760753.676469:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742932637905 time 742970334066 state 0x0 ref 1 00000100:00100000:3.0:1789760753.676484:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742970594482 time 742970334066 state 0x1 ref 1 00000100:00100000:3.0:1789760753.676512:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742970594482 time 743007443656 state 0x0 ref 1 00000100:00100000:3.0:1789760753.676524:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484265984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760753.676572:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22376448, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:3.0:1789760753.676582:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.676598:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:3.0:1789760753.676602:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760753.676608:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22376448->22380543] (req 22376448->22380543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.676621:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22376448->22380543] (req 22376448->22380543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760753.721415:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972781 is committed 00002000:00010000:1.0:1789760753.721520:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22376448 -> 22380544 00010000:00010000:1.0:1789760753.721534:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22376448->22380543] (req 22376448->22380543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.721547:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22376448->22380543] (req 22376448->22380543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.721557:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22376448->22380543] (req 22376448->22380543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.721563:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760753.721568:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22376448->22380543] (req 22376448->22380543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.721578:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22376448->22380543] (req 22376448->22380543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.721655:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22376448->22380543] (req 22376448->22380543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.721663:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396208f1e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22376448->22380543] (req 22376448->22380543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760753.721794:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484265984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45267us (45877us total) trans 4294972781 rc 0/0 00000100:00100000:1.0:1789760753.721804:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5492 00000100:00100000:1.0:1789760753.721814:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742970594482 time 743007443656 state 0x2 ref 0 00000100:00100000:0.0:1789760753.724933:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760753.727148:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484266112 00000100:00100000:2.0:1789760753.727162:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484266112 00000100:00100000:2.0:1789760753.727235:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 742970594482 time 743007443656 state 0x0 ref 1 00000100:00100000:2.0:1789760753.727251:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743007704072 time 743007443656 state 0x1 ref 1 00000100:00100000:2.0:1789760753.727396:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743007704072 time 743058325627 state 0x0 ref 1 00000100:00100000:2.0:1789760753.727411:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484266112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760753.727462:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22380544, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:2.0:1789760753.727471:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.727488:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:2.0:1789760753.727492:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760753.727498:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22380544->22384639] (req 22380544->22384639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.727512:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22380544->22384639] (req 22380544->22384639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760753.760863:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22380544 -> 22384640 00010000:00010000:3.0:1789760753.760882:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22380544->22384639] (req 22380544->22384639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.760895:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22380544->22384639] (req 22380544->22384639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.760910:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22380544->22384639] (req 22380544->22384639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.760917:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760753.760922:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22380544->22384639] (req 22380544->22384639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.760931:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22380544->22384639] (req 22380544->22384639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.760950:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22380544->22384639] (req 22380544->22384639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.760958:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396208f25 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22380544->22384639] (req 22380544->22384639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760753.761155:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972782 is committed 00000100:00100000:3.0:1789760753.761249:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484266112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33837us (36321us total) trans 4294972782 rc 0/0 00000100:00100000:3.0:1789760753.761268:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5493 00000100:00100000:3.0:1789760753.761278:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743007704072 time 743058325627 state 0x2 ref 0 00000100:00100000:0.0:1789760753.767780:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760753.767871:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484266240 00000100:00100000:1.0:1789760753.767881:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484266240 00000100:00100000:1.0:1789760753.767938:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743007704072 time 743058325627 state 0x0 ref 1 00000100:00100000:1.0:1789760753.767958:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743058586043 time 743058325627 state 0x1 ref 1 00000100:00100000:1.0:1789760753.767990:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743058586043 time 743098920276 state 0x0 ref 1 00000100:00100000:1.0:1789760753.768011:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484266240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760753.768061:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22384640, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:1.0:1789760753.768102:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.768118:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:1.0:1789760753.768122:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760753.768128:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22384640->22388735] (req 22384640->22388735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.768141:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22384640->22388735] (req 22384640->22388735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760753.816088:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22384640 -> 22388736 00010000:00010000:1.0:1789760753.816107:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22384640->22388735] (req 22384640->22388735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.816120:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22384640->22388735] (req 22384640->22388735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.816130:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22384640->22388735] (req 22384640->22388735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.816142:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760753.816147:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22384640->22388735] (req 22384640->22388735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.816155:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22384640->22388735] (req 22384640->22388735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.816173:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22384640->22388735] (req 22384640->22388735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.816181:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396208f2c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22384640->22388735] (req 22384640->22388735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760753.816272:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484266240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48267us (48497us total) trans 4294972783 rc 0/0 00000100:00100000:1.0:1789760753.816282:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5494 00000100:00100000:1.0:1789760753.816292:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743058586043 time 743098920276 state 0x2 ref 0 00000001:00080000:3.0:1789760753.817852:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972783 is committed 00000100:00100000:1.0:1789760753.821545:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760753.821637:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484266368 00000100:00100000:0.0:1789760753.821646:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484266368 00000100:00100000:0.0:1789760753.821706:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743058586043 time 743098920276 state 0x0 ref 1 00000100:00100000:0.0:1789760753.821720:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743099180692 time 743098920276 state 0x1 ref 1 00000100:00100000:0.0:1789760753.821752:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743099180692 time 743152683201 state 0x0 ref 1 00000100:00100000:0.0:1789760753.821771:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484266368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760753.821818:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22388736, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:0.0:1789760753.821828:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.821843:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:0.0:1789760753.821848:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760753.821853:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22388736->22392831] (req 22388736->22392831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.821867:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22388736->22392831] (req 22388736->22392831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760753.852616:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972784 is committed 00002000:00010000:3.0:1789760753.852766:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22388736 -> 22392832 00010000:00010000:3.0:1789760753.852783:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22388736->22392831] (req 22388736->22392831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.852801:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22388736->22392831] (req 22388736->22392831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.852811:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22388736->22392831] (req 22388736->22392831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.852819:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760753.852824:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22388736->22392831] (req 22388736->22392831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.852833:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22388736->22392831] (req 22388736->22392831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.852852:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22388736->22392831] (req 22388736->22392831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.852859:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396208f33 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22388736->22392831] (req 22388736->22392831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760753.852963:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484266368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31190us (31420us total) trans 4294972784 rc 0/0 00000100:00100000:3.0:1789760753.852973:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5495 00000100:00100000:3.0:1789760753.852983:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743099180692 time 743152683201 state 0x2 ref 0 00000100:00100000:0.0:1789760753.858890:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760753.859835:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484266496 00000100:00100000:3.0:1789760753.859846:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484266496 00000100:00100000:2.0:1789760753.861713:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743099180692 time 743152683201 state 0x0 ref 1 00000100:00100000:2.0:1789760753.861731:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743152943617 time 743152683201 state 0x1 ref 1 00000100:00100000:2.0:1789760753.861766:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743152943617 time 743192697273 state 0x0 ref 1 00000100:00100000:2.0:1789760753.861779:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484266496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760753.861824:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22392832, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:2.0:1789760753.861833:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.861850:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:2.0:1789760753.861854:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760753.861860:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22392832->22396927] (req 22392832->22396927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760753.861873:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22392832->22396927] (req 22392832->22396927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760753.900230:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972785 is committed 00002000:00010000:1.0:1789760753.900330:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22392832 -> 22396928 00010000:00010000:1.0:1789760753.900345:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22392832->22396927] (req 22392832->22396927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.900357:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22392832->22396927] (req 22392832->22396927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.900367:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22392832->22396927] (req 22392832->22396927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.900374:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760753.900378:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22392832->22396927] (req 22392832->22396927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.900386:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22392832->22396927] (req 22392832->22396927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.900404:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22392832->22396927] (req 22392832->22396927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.900411:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396208f3a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22392832->22396927] (req 22392832->22396927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760753.900600:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484266496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38817us (41713us total) trans 4294972785 rc 0/0 00000100:00100000:1.0:1789760753.900610:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5496 00000100:00100000:1.0:1789760753.900620:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743152943617 time 743192697273 state 0x2 ref 0 00000100:00100000:1.0:1789760753.904833:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760753.904904:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484266624 00000100:00100000:3.0:1789760753.904913:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484266624 00000100:00100000:3.0:1789760753.904967:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743152943617 time 743192697273 state 0x0 ref 1 00000100:00100000:3.0:1789760753.904984:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743192957689 time 743192697273 state 0x1 ref 1 00000100:00100000:3.0:1789760753.905023:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743192957689 time 743235954291 state 0x0 ref 1 00000100:00100000:3.0:1789760753.905036:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706f450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484266624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760753.905117:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22396928, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:3.0:1789760753.905126:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.905142:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:3.0:1789760753.905147:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760753.905153:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22396928->22401023] (req 22396928->22401023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.905169:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22396928->22401023] (req 22396928->22401023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760753.954322:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22396928 -> 22401024 00010000:00010000:3.0:1789760753.954342:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22396928->22401023] (req 22396928->22401023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.954356:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22396928->22401023] (req 22396928->22401023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.954366:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22396928->22401023] (req 22396928->22401023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.954374:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760753.954379:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22396928->22401023] (req 22396928->22401023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.954389:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22396928->22401023] (req 22396928->22401023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.954408:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22396928->22401023] (req 22396928->22401023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760753.954416:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f41 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22396928->22401023] (req 22396928->22401023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760753.954503:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706f450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484266624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49464us (49675us total) trans 4294972786 rc 0/0 00000100:00100000:3.0:1789760753.954513:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5497 00000100:00100000:3.0:1789760753.954522:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743192957689 time 743235954291 state 0x2 ref 0 00000001:00080000:1.0:1789760753.959373:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972786 is committed 00000100:00100000:2.0:1789760753.959570:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760753.959741:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484266752 00000100:00100000:1.0:1789760753.959751:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484266752 00000100:00100000:1.0:1789760753.959799:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743192957689 time 743235954291 state 0x0 ref 1 00000100:00100000:1.0:1789760753.959852:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743236214707 time 743235954291 state 0x1 ref 1 00000100:00100000:1.0:1789760753.959888:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743236214707 time 743290818906 state 0x0 ref 1 00000100:00100000:1.0:1789760753.959900:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484266752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760753.959947:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22401024, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:1.0:1789760753.959955:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.959970:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:1.0:1789760753.959974:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760753.959980:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22401024->22405119] (req 22401024->22405119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760753.959994:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22401024->22405119] (req 22401024->22405119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760753.997733:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22401024 -> 22405120 00010000:00010000:0.0:1789760753.997753:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22401024->22405119] (req 22401024->22405119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.997766:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22401024->22405119] (req 22401024->22405119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.997775:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22401024->22405119] (req 22401024->22405119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.997783:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760753.997787:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22401024->22405119] (req 22401024->22405119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.997801:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22401024->22405119] (req 22401024->22405119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.997820:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22401024->22405119] (req 22401024->22405119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760753.997829:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396208f48 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22401024->22405119] (req 22401024->22405119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760753.997931:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484266752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38029us (38360us total) trans 4294972787 rc 0/0 00000100:00100000:0.0:1789760753.997941:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5498 00000100:00100000:0.0:1789760753.997951:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743236214707 time 743290818906 state 0x2 ref 0 00000001:00080000:2.0:1789760754.001322:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972787 is committed 00000100:00100000:3.0:1789760754.002622:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760754.002900:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484266880 00000100:00100000:2.0:1789760754.002909:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484266880 00000100:00100000:2.0:1789760754.002980:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743236214707 time 743290818906 state 0x0 ref 1 00000100:00100000:2.0:1789760754.002996:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743291079322 time 743290818906 state 0x1 ref 1 00000100:00100000:2.0:1789760754.003039:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743291079322 time 743333970197 state 0x0 ref 1 00000100:00100000:2.0:1789760754.003052:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484266880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760754.003131:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22405120, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:2.0:1789760754.003139:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.003155:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:2.0:1789760754.003159:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760754.003164:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22405120->22409215] (req 22405120->22409215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.003178:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22405120->22409215] (req 22405120->22409215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760754.003786:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:1.0:1789760754.031993:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22405120 -> 22409216 00010000:00010000:1.0:1789760754.032013:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22405120->22409215] (req 22405120->22409215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.032026:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22405120->22409215] (req 22405120->22409215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.032034:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22405120->22409215] (req 22405120->22409215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.032041:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.032046:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22405120->22409215] (req 22405120->22409215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.032054:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22405120->22409215] (req 22405120->22409215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.032088:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22405120->22409215] (req 22405120->22409215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.032096:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396208f4f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22405120->22409215] (req 22405120->22409215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.032214:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484266880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29159us (29593us total) trans 4294972788 rc 0/0 00000100:00100000:1.0:1789760754.032233:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5499 00000100:00100000:1.0:1789760754.032243:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743291079322 time 743333970197 state 0x2 ref 0 00000001:00080000:2.0:1789760754.032323:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972788 is committed 00000100:00100000:3.0:1789760754.037477:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760754.037805:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484267008 00000100:00100000:2.0:1789760754.037815:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484267008 00000100:00100000:2.0:1789760754.037944:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743291079322 time 743333970197 state 0x0 ref 1 00000100:00100000:2.0:1789760754.037961:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743334230613 time 743333970197 state 0x1 ref 1 00000100:00100000:2.0:1789760754.037994:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743334230613 time 743368924877 state 0x0 ref 1 00000100:00100000:2.0:1789760754.038007:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484267008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760754.038142:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22409216, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:2.0:1789760754.038151:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.038167:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:2.0:1789760754.038171:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760754.038176:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22409216->22413311] (req 22409216->22413311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.038190:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22409216->22413311] (req 22409216->22413311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.061291:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501ba0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508926976:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760754.061365:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760754.061486:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508926976 00000100:00100000:3.0:1789760754.061494:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508926976 00000100:00100000:3.0:1789760754.061544:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 450 00000100:00100000:3.0:1789760754.061554:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508926976:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:3.0:1789760754.061701:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:3.0:1789760754.061758:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508926976:12345-0@lo:41:osp-pre-1-0.0 Request processed in 205us (392us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760754.061765:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 450 00000100:00100000:1.0:1789760754.061853:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501ba0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508926976:0@lo:41:osp-pre-1-0.0 00002000:00010000:1.0:1789760754.092425:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22409216 -> 22413312 00010000:00010000:1.0:1789760754.092445:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22409216->22413311] (req 22409216->22413311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.092461:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22409216->22413311] (req 22409216->22413311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.092468:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22409216->22413311] (req 22409216->22413311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.092475:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.092479:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22409216->22413311] (req 22409216->22413311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.092488:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22409216->22413311] (req 22409216->22413311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.092508:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22409216->22413311] (req 22409216->22413311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.092517:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396208f56 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22409216->22413311] (req 22409216->22413311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.092616:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484267008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54607us (55145us total) trans 4294972789 rc 0/0 00000100:00100000:1.0:1789760754.092626:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5500 00000100:00100000:1.0:1789760754.092635:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743334230613 time 743368924877 state 0x2 ref 0 00000001:00080000:2.0:1789760754.094589:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972789 is committed 00000100:00100000:3.0:1789760754.095921:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.096137:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484267136 00000100:00100000:1.0:1789760754.096147:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484267136 00000100:00100000:1.0:1789760754.096199:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743334230613 time 743368924877 state 0x0 ref 1 00000100:00100000:1.0:1789760754.096213:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743369185293 time 743368924877 state 0x1 ref 1 00000100:00100000:1.0:1789760754.096246:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743369185293 time 743427176396 state 0x0 ref 1 00000100:00100000:1.0:1789760754.096260:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484267136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.096317:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22413312, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:1.0:1789760754.096327:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.096344:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:1.0:1789760754.096348:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.096355:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22413312->22417407] (req 22413312->22417407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.096369:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22413312->22417407] (req 22413312->22417407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760754.137553:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22413312 -> 22417408 00010000:00010000:1.0:1789760754.137579:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22413312->22417407] (req 22413312->22417407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.137602:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22413312->22417407] (req 22413312->22417407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.137617:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22413312->22417407] (req 22413312->22417407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.137625:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.137630:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22413312->22417407] (req 22413312->22417407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.137639:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22413312->22417407] (req 22413312->22417407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.137661:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22413312->22417407] (req 22413312->22417407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.137670:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f5d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22413312->22417407] (req 22413312->22417407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.137767:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484267136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41503us (41848us total) trans 4294972790 rc 0/0 00000100:00100000:1.0:1789760754.137778:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5501 00000100:00100000:1.0:1789760754.137787:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743369185293 time 743427176396 state 0x2 ref 0 00000001:00080000:2.0:1789760754.140167:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972790 is committed 00000100:00100000:2.0:1789760754.142880:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760754.142981:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484267264 00000100:00100000:0.0:1789760754.142992:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484267264 00000100:00100000:0.0:1789760754.143231:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743369185293 time 743427176396 state 0x0 ref 1 00000100:00100000:0.0:1789760754.143248:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743427436812 time 743427176396 state 0x1 ref 1 00000100:00100000:0.0:1789760754.143370:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743427436812 time 743474236906 state 0x0 ref 1 00000100:00100000:0.0:1789760754.143388:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484267264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760754.143436:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22417408, mtime: 0x6aad94f1, atime: 0x0, ctime: 0x6aad94f1, blocks: 0xb007 00010000:00010000:0.0:1789760754.143445:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.143460:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:0.0:1789760754.143464:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760754.143471:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22417408->22421503] (req 22417408->22421503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.143484:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22417408->22421503] (req 22417408->22421503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760754.175892:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972791 is committed 00002000:00010000:0.0:1789760754.175932:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22417408 -> 22421504 00002000:00010000:0.0:1789760754.175937:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760753 -> 1789760754 00002000:00010000:0.0:1789760754.175939:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760753 -> 1789760754 00010000:00010000:0.0:1789760754.175952:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22417408->22421503] (req 22417408->22421503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.175965:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22417408->22421503] (req 22417408->22421503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.175974:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22417408->22421503] (req 22417408->22421503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.175981:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760754.175985:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22417408->22421503] (req 22417408->22421503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.175995:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22417408->22421503] (req 22417408->22421503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.176026:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22417408->22421503] (req 22417408->22421503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.176033:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f64 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22417408->22421503] (req 22417408->22421503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760754.176166:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484267264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32778us (33291us total) trans 4294972791 rc 0/0 00000100:00100000:0.0:1789760754.176177:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5502 00000100:00100000:0.0:1789760754.176186:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743427436812 time 743474236906 state 0x2 ref 0 00000100:00100000:0.0:1789760754.181251:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760754.181393:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484267392 00000100:00100000:2.0:1789760754.181403:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484267392 00000100:00100000:2.0:1789760754.181457:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743427436812 time 743474236906 state 0x0 ref 1 00000100:00100000:2.0:1789760754.181471:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743474497322 time 743474236906 state 0x1 ref 1 00000100:00100000:2.0:1789760754.181536:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743474497322 time 743512467484 state 0x0 ref 1 00000100:00100000:2.0:1789760754.181552:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484267392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760754.181677:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22421504, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:2.0:1789760754.181686:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.181701:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:2.0:1789760754.181706:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760754.181712:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22421504->22425599] (req 22421504->22425599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.181726:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22421504->22425599] (req 22421504->22425599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760754.216994:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972792 is committed 00002000:00010000:2.0:1789760754.217199:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22421504 -> 22425600 00010000:00010000:2.0:1789760754.217215:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22421504->22425599] (req 22421504->22425599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.217240:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22421504->22425599] (req 22421504->22425599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.217250:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22421504->22425599] (req 22421504->22425599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.217257:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760754.217263:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22421504->22425599] (req 22421504->22425599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.217274:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22421504->22425599] (req 22421504->22425599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.217295:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22421504->22425599] (req 22421504->22425599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.217302:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396208f6b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22421504->22425599] (req 22421504->22425599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760754.217439:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484267392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35884us (36191us total) trans 4294972792 rc 0/0 00000100:00100000:2.0:1789760754.217449:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5503 00000100:00100000:2.0:1789760754.217459:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743474497322 time 743512467484 state 0x2 ref 0 00000100:00100000:2.0:1789760754.221197:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.221284:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484267520 00000100:00100000:1.0:1789760754.221294:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484267520 00000100:00100000:1.0:1789760754.221355:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743474497322 time 743512467484 state 0x0 ref 1 00000100:00100000:1.0:1789760754.221372:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743512727900 time 743512467484 state 0x1 ref 1 00000100:00100000:1.0:1789760754.221401:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743512727900 time 743552332662 state 0x0 ref 1 00000100:00100000:1.0:1789760754.221412:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484267520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.221455:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22425600, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760754.221462:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.221477:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:1.0:1789760754.221481:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.221487:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22425600->22429695] (req 22425600->22429695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.221509:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22425600->22429695] (req 22425600->22429695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760754.244147:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22425600 -> 22429696 00010000:00010000:2.0:1789760754.244165:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22425600->22429695] (req 22425600->22429695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.244176:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22425600->22429695] (req 22425600->22429695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.244182:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22425600->22429695] (req 22425600->22429695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.244191:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760754.244194:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22425600->22429695] (req 22425600->22429695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.244201:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22425600->22429695] (req 22425600->22429695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.244217:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22425600->22429695] (req 22425600->22429695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.244234:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396208f72 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22425600->22429695] (req 22425600->22429695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760754.244336:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484267520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 22919us (23143us total) trans 4294972793 rc 0/0 00000100:00100000:2.0:1789760754.244347:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5504 00000100:00100000:2.0:1789760754.244357:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743512727900 time 743552332662 state 0x2 ref 0 00000001:00080000:3.0:1789760754.244872:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972793 is committed 00000100:00100000:0.0:1789760754.247938:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.248218:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484267648 00000100:00100000:1.0:1789760754.248237:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484267648 00000100:00100000:0.0:1789760754.248426:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743512727900 time 743552332662 state 0x0 ref 1 00000100:00100000:0.0:1789760754.248441:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743552593078 time 743552332662 state 0x1 ref 1 00000100:00100000:0.0:1789760754.248682:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743552593078 time 743579611585 state 0x0 ref 1 00000100:00100000:0.0:1789760754.248697:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484267648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760754.248748:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22429696, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:0.0:1789760754.248757:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.248772:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:0.0:1789760754.248777:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760754.248783:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22429696->22433791] (req 22429696->22433791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.248797:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22429696->22433791] (req 22429696->22433791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760754.278770:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972794 is committed 00002000:00010000:0.0:1789760754.278857:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22429696 -> 22433792 00010000:00010000:0.0:1789760754.278932:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22429696->22433791] (req 22429696->22433791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.278946:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22429696->22433791] (req 22429696->22433791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.278954:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22429696->22433791] (req 22429696->22433791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.278961:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760754.278967:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22429696->22433791] (req 22429696->22433791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.278976:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22429696->22433791] (req 22429696->22433791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.278997:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22429696->22433791] (req 22429696->22433791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.279089:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f79 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22429696->22433791] (req 22429696->22433791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760754.279234:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484267648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30528us (31292us total) trans 4294972794 rc 0/0 00000100:00100000:0.0:1789760754.279244:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5505 00000100:00100000:0.0:1789760754.279254:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743552593078 time 743579611585 state 0x2 ref 0 00000100:00100000:1.0:1789760754.282412:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760754.282503:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484267776 00000100:00100000:3.0:1789760754.282514:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484267776 00000100:00100000:3.0:1789760754.282570:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743552593078 time 743579611585 state 0x0 ref 1 00000100:00100000:3.0:1789760754.282586:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743579872001 time 743579611585 state 0x1 ref 1 00000100:00100000:3.0:1789760754.282617:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743579872001 time 743613547968 state 0x0 ref 1 00000100:00100000:3.0:1789760754.282629:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484267776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760754.282677:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22433792, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:3.0:1789760754.282685:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.282699:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:3.0:1789760754.282703:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760754.282710:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22433792->22437887] (req 22433792->22437887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.282730:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22433792->22437887] (req 22433792->22437887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760754.319329:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22433792 -> 22437888 00010000:00010000:1.0:1789760754.319350:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22433792->22437887] (req 22433792->22437887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.319364:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22433792->22437887] (req 22433792->22437887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.319374:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22433792->22437887] (req 22433792->22437887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.319383:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.319387:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22433792->22437887] (req 22433792->22437887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.319443:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22433792->22437887] (req 22433792->22437887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.319468:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22433792->22437887] (req 22433792->22437887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.319476:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396208f80 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22433792->22437887] (req 22433792->22437887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.319573:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484267776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36941us (37166us total) trans 4294972795 rc 0/0 00000100:00100000:1.0:1789760754.319584:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5506 00000100:00100000:1.0:1789760754.319593:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743579872001 time 743613547968 state 0x2 ref 0 00000001:00080000:2.0:1789760754.319675:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972795 is committed 00000100:00100000:2.0:1789760754.325693:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.325828:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484267904 00000100:00100000:1.0:1789760754.325838:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484267904 00000100:00100000:1.0:1789760754.325894:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743579872001 time 743613547968 state 0x0 ref 1 00000100:00100000:1.0:1789760754.325910:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743613808384 time 743613547968 state 0x1 ref 1 00000100:00100000:1.0:1789760754.325934:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743613808384 time 743656864937 state 0x0 ref 1 00000100:00100000:1.0:1789760754.325945:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484267904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.325998:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22437888, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760754.326007:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.326024:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:1.0:1789760754.326029:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.326035:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22437888->22441983] (req 22437888->22441983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.326052:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22437888->22441983] (req 22437888->22441983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760754.349351:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972796 is committed 00002000:00010000:3.0:1789760754.349578:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22437888 -> 22441984 00010000:00010000:3.0:1789760754.349594:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22437888->22441983] (req 22437888->22441983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.349608:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22437888->22441983] (req 22437888->22441983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.349617:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22437888->22441983] (req 22437888->22441983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.349624:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760754.349629:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22437888->22441983] (req 22437888->22441983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.349638:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22437888->22441983] (req 22437888->22441983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.349656:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22437888->22441983] (req 22437888->22441983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.349664:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396208f87 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22437888->22441983] (req 22437888->22441983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760754.349767:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484267904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23818us (24079us total) trans 4294972796 rc 0/0 00000100:00100000:3.0:1789760754.349778:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5507 00000100:00100000:3.0:1789760754.349787:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743613808384 time 743656864937 state 0x2 ref 0 00000100:00100000:3.0:1789760754.353017:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.353251:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484268032 00000100:00100000:1.0:1789760754.353262:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484268032 00000100:00100000:1.0:1789760754.353317:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743613808384 time 743656864937 state 0x0 ref 1 00000100:00100000:1.0:1789760754.353331:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743657125353 time 743656864937 state 0x1 ref 1 00000100:00100000:1.0:1789760754.353355:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743657125353 time 743684286222 state 0x0 ref 1 00000100:00100000:1.0:1789760754.353367:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484268032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.353415:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22441984, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760754.353423:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.353439:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:1.0:1789760754.353444:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.353449:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22441984->22446079] (req 22441984->22446079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.353463:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22441984->22446079] (req 22441984->22446079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760754.387921:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972797 is committed 00002000:00010000:1.0:1789760754.388127:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22441984 -> 22446080 00010000:00010000:1.0:1789760754.388143:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22441984->22446079] (req 22441984->22446079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.388156:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22441984->22446079] (req 22441984->22446079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.388165:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22441984->22446079] (req 22441984->22446079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.388173:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.388178:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22441984->22446079] (req 22441984->22446079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.388188:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22441984->22446079] (req 22441984->22446079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.388210:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22441984->22446079] (req 22441984->22446079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.388226:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208f8e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22441984->22446079] (req 22441984->22446079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.388520:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484268032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35150us (35513us total) trans 4294972797 rc 0/0 00000100:00100000:1.0:1789760754.388531:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5508 00000100:00100000:1.0:1789760754.388540:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743657125353 time 743684286222 state 0x2 ref 0 00000100:00100000:1.0:1789760754.391882:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760754.392038:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484268160 00000100:00100000:0.0:1789760754.392048:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484268160 00000100:00100000:0.0:1789760754.392136:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743657125353 time 743684286222 state 0x0 ref 1 00000100:00100000:0.0:1789760754.392152:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743684546638 time 743684286222 state 0x1 ref 1 00000100:00100000:0.0:1789760754.392183:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743684546638 time 743723113719 state 0x0 ref 1 00000100:00100000:0.0:1789760754.392197:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484268160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760754.392254:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22446080, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:0.0:1789760754.392263:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.392279:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:0.0:1789760754.392283:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760754.392290:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22446080->22450175] (req 22446080->22450175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.392303:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22446080->22450175] (req 22446080->22450175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760754.427739:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972798 is committed 00002000:00010000:0.0:1789760754.427834:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22446080 -> 22450176 00010000:00010000:0.0:1789760754.427850:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22446080->22450175] (req 22446080->22450175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.427864:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22446080->22450175] (req 22446080->22450175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.427874:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22446080->22450175] (req 22446080->22450175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.427881:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760754.427897:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22446080->22450175] (req 22446080->22450175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.427907:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22446080->22450175] (req 22446080->22450175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.427927:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22446080->22450175] (req 22446080->22450175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.427935:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208f95 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22446080->22450175] (req 22446080->22450175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760754.428038:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484268160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35839us (36160us total) trans 4294972798 rc 0/0 00000100:00100000:0.0:1789760754.428050:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5509 00000100:00100000:0.0:1789760754.428060:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743684546638 time 743723113719 state 0x2 ref 0 00000100:00100000:2.0:1789760754.431894:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760754.432388:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484268288 00000100:00100000:3.0:1789760754.432397:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484268288 00000100:00100000:3.0:1789760754.432452:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743684546638 time 743723113719 state 0x0 ref 1 00000100:00100000:3.0:1789760754.432466:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743723374135 time 743723113719 state 0x1 ref 1 00000100:00100000:3.0:1789760754.432495:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743723374135 time 743763425936 state 0x0 ref 1 00000100:00100000:3.0:1789760754.432508:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484268288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760754.432557:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22450176, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:3.0:1789760754.432566:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.433411:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:3.0:1789760754.433419:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760754.433426:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22450176->22454271] (req 22450176->22454271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.433448:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22450176->22454271] (req 22450176->22454271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760754.467318:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22450176 -> 22454272 00010000:00010000:1.0:1789760754.467340:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22450176->22454271] (req 22450176->22454271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.467353:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22450176->22454271] (req 22450176->22454271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.467362:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22450176->22454271] (req 22450176->22454271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.467368:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.467373:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22450176->22454271] (req 22450176->22454271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.467386:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22450176->22454271] (req 22450176->22454271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.467405:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22450176->22454271] (req 22450176->22454271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.467411:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396208f9c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22450176->22454271] (req 22450176->22454271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.467512:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484268288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35002us (35622us total) trans 4294972799 rc 0/0 00000100:00100000:1.0:1789760754.467524:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5510 00000100:00100000:1.0:1789760754.467534:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743723374135 time 743763425936 state 0x2 ref 0 00000001:00080000:2.0:1789760754.469391:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972799 is committed 00000100:00100000:3.0:1789760754.473895:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.475031:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484268416 00000100:00100000:1.0:1789760754.475043:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484268416 00000100:00100000:1.0:1789760754.475116:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743723374135 time 743763425936 state 0x0 ref 1 00000100:00100000:1.0:1789760754.475133:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743763686352 time 743763425936 state 0x1 ref 1 00000100:00100000:1.0:1789760754.475166:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743763686352 time 743806097599 state 0x0 ref 1 00000100:00100000:1.0:1789760754.475178:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484268416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.475241:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22454272, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760754.475251:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.475268:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:1.0:1789760754.475272:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.475278:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22454272->22458367] (req 22454272->22458367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.475292:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22454272->22458367] (req 22454272->22458367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760754.505709:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972800 is committed 00002000:00010000:1.0:1789760754.505795:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22454272 -> 22458368 00010000:00010000:1.0:1789760754.505811:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22454272->22458367] (req 22454272->22458367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.505826:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22454272->22458367] (req 22454272->22458367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.505835:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22454272->22458367] (req 22454272->22458367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.505843:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.505848:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22454272->22458367] (req 22454272->22458367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.505857:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22454272->22458367] (req 22454272->22458367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.505877:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22454272->22458367] (req 22454272->22458367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.505885:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fa3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22454272->22458367] (req 22454272->22458367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.506194:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484268416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31012us (32304us total) trans 4294972800 rc 0/0 00000100:00100000:1.0:1789760754.506205:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5511 00000100:00100000:1.0:1789760754.506215:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743763686352 time 743806097599 state 0x2 ref 0 00000100:00100000:3.0:1789760754.510023:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.510153:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484268544 00000100:00100000:1.0:1789760754.510162:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484268544 00000100:00100000:1.0:1789760754.510215:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743763686352 time 743806097599 state 0x0 ref 1 00000100:00100000:1.0:1789760754.510240:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743806358015 time 743806097599 state 0x1 ref 1 00000100:00100000:1.0:1789760754.510273:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743806358015 time 743841203851 state 0x0 ref 1 00000100:00100000:1.0:1789760754.510286:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484268544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.510345:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22458368, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760754.510354:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.510370:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:1.0:1789760754.510375:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.510382:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22458368->22462463] (req 22458368->22462463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.510401:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22458368->22462463] (req 22458368->22462463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760754.544789:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972801 is committed 00002000:00010000:1.0:1789760754.544896:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22458368 -> 22462464 00010000:00010000:1.0:1789760754.544910:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22458368->22462463] (req 22458368->22462463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.544922:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22458368->22462463] (req 22458368->22462463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.544931:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22458368->22462463] (req 22458368->22462463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.544938:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.544942:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22458368->22462463] (req 22458368->22462463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.544950:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22458368->22462463] (req 22458368->22462463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.544968:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22458368->22462463] (req 22458368->22462463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.544975:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208faa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22458368->22462463] (req 22458368->22462463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.545123:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484268544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34833us (35103us total) trans 4294972801 rc 0/0 00000100:00100000:1.0:1789760754.545133:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5512 00000100:00100000:1.0:1789760754.545769:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743806358015 time 743841203851 state 0x2 ref 0 00000100:00100000:2.0:1789760754.552265:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760754.555567:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484268672 00000100:00100000:0.0:1789760754.555581:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484268672 00000100:00100000:0.0:1789760754.555651:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743806358015 time 743841203851 state 0x0 ref 1 00000100:00100000:0.0:1789760754.555668:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743841464267 time 743841203851 state 0x1 ref 1 00000100:00100000:0.0:1789760754.555702:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743841464267 time 743886632423 state 0x0 ref 1 00000100:00100000:0.0:1789760754.555716:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484268672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760754.555764:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22462464, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:0.0:1789760754.555774:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.555790:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:0.0:1789760754.555795:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760754.555802:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22462464->22466559] (req 22462464->22466559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.555815:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22462464->22466559] (req 22462464->22466559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.573493:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760754.612538:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972802 is committed 00002000:00010000:0.0:1789760754.613304:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22462464 -> 22466560 00010000:00010000:0.0:1789760754.613321:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22462464->22466559] (req 22462464->22466559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.613334:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22462464->22466559] (req 22462464->22466559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.613343:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22462464->22466559] (req 22462464->22466559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.613351:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760754.613355:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22462464->22466559] (req 22462464->22466559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.613365:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22462464->22466559] (req 22462464->22466559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.613385:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22462464->22466559] (req 22462464->22466559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.613394:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fb1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22462464->22466559] (req 22462464->22466559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760754.613548:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484268672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57830us (61287us total) trans 4294972802 rc 0/0 00000100:00100000:0.0:1789760754.613559:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5513 00000100:00100000:0.0:1789760754.613569:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743841464267 time 743886632423 state 0x2 ref 0 00000100:00100000:3.0:1789760754.617825:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.617976:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484268800 00000100:00100000:1.0:1789760754.617986:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484268800 00000100:00100000:1.0:1789760754.618322:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743841464267 time 743886632423 state 0x0 ref 1 00000100:00100000:1.0:1789760754.618338:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743886892839 time 743886632423 state 0x1 ref 1 00000100:00100000:0.0:1789760754.618490:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743886892839 time 743949418314 state 0x0 ref 1 00000100:00100000:0.0:1789760754.618512:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484268800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760754.618566:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22466560, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:0.0:1789760754.618574:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.618589:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:0.0:1789760754.618593:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760754.618598:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22466560->22470655] (req 22466560->22470655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.618610:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22466560->22470655] (req 22466560->22470655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760754.651417:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972803 is committed 00002000:00010000:1.0:1789760754.651594:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22466560 -> 22470656 00010000:00010000:1.0:1789760754.651610:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22466560->22470655] (req 22466560->22470655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.651622:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22466560->22470655] (req 22466560->22470655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.651632:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22466560->22470655] (req 22466560->22470655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.651638:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.651644:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22466560->22470655] (req 22466560->22470655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.651659:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22466560->22470655] (req 22466560->22470655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.651678:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22466560->22470655] (req 22466560->22470655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.651686:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208fb8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22466560->22470655] (req 22466560->22470655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.651791:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484268800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33285us (33971us total) trans 4294972803 rc 0/0 00000100:00100000:1.0:1789760754.651802:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5514 00000100:00100000:1.0:1789760754.651812:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743886892839 time 743949418314 state 0x2 ref 0 00000100:00100000:3.0:1789760754.659250:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.659341:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484268928 00000100:00100000:1.0:1789760754.659351:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484268928 00000100:00100000:1.0:1789760754.659406:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743886892839 time 743949418314 state 0x0 ref 1 00000100:00100000:1.0:1789760754.659421:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743949678730 time 743949418314 state 0x1 ref 1 00000100:00100000:1.0:1789760754.659453:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743949678730 time 743990383847 state 0x0 ref 1 00000100:00100000:1.0:1789760754.659466:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484268928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.659511:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22470656, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760754.659520:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.659535:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=743s v=(0 0) 00010000:00010000:1.0:1789760754.659539:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.659546:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22470656->22474751] (req 22470656->22474751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.659559:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22470656->22474751] (req 22470656->22474751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760754.698354:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972804 is committed 00002000:00010000:1.0:1789760754.698511:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22470656 -> 22474752 00010000:00010000:1.0:1789760754.698526:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22470656->22474751] (req 22470656->22474751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.698540:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22470656->22474751] (req 22470656->22474751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.698549:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22470656->22474751] (req 22470656->22474751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.698557:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.698561:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22470656->22474751] (req 22470656->22474751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.698570:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22470656->22474751] (req 22470656->22474751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.698592:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22470656->22474751] (req 22470656->22474751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.698601:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fbf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22470656->22474751] (req 22470656->22474751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.698691:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484268928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39223us (39447us total) trans 4294972804 rc 0/0 00000100:00100000:1.0:1789760754.698702:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5515 00000100:00100000:1.0:1789760754.698712:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743949678730 time 743990383847 state 0x2 ref 0 00000100:00100000:3.0:1789760754.706892:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760754.707208:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484269056 00000100:00100000:2.0:1789760754.707227:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484269056 00000100:00100000:2.0:1789760754.707277:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743949678730 time 743990383847 state 0x0 ref 1 00000100:00100000:2.0:1789760754.707292:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743990644263 time 743990383847 state 0x1 ref 1 00000100:00100000:2.0:1789760754.707417:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743990644263 time 744038346606 state 0x0 ref 1 00000100:00100000:2.0:1789760754.707430:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484269056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760754.707478:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22474752, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:2.0:1789760754.707488:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.707504:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:2.0:1789760754.707509:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760754.707517:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22474752->22478847] (req 22474752->22478847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.707529:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22474752->22478847] (req 22474752->22478847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760754.748725:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972805 is committed 00002000:00010000:1.0:1789760754.749315:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22474752 -> 22478848 00010000:00010000:1.0:1789760754.749330:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22474752->22478847] (req 22474752->22478847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.749343:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22474752->22478847] (req 22474752->22478847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.749352:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22474752->22478847] (req 22474752->22478847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.749360:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.749365:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22474752->22478847] (req 22474752->22478847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.749375:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22474752->22478847] (req 22474752->22478847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.749399:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22474752->22478847] (req 22474752->22478847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.749407:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396208fc6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22474752->22478847] (req 22474752->22478847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.749511:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484269056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42078us (42626us total) trans 4294972805 rc 0/0 00000100:00100000:1.0:1789760754.749521:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5516 00000100:00100000:1.0:1789760754.749532:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743990644263 time 744038346606 state 0x2 ref 0 00000100:00100000:3.0:1789760754.753844:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760754.754283:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484269184 00000100:00100000:0.0:1789760754.754294:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484269184 00000100:00100000:0.0:1789760754.754355:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 743990644263 time 744038346606 state 0x0 ref 1 00000100:00100000:0.0:1789760754.754370:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744038607022 time 744038346606 state 0x1 ref 1 00000100:00100000:0.0:1789760754.754395:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744038607022 time 744085326490 state 0x0 ref 1 00000100:00100000:0.0:1789760754.754409:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484269184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760754.754458:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22478848, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:0.0:1789760754.754466:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.754482:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:0.0:1789760754.754487:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760754.754493:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22478848->22482943] (req 22478848->22482943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.754506:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22478848->22482943] (req 22478848->22482943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760754.812717:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972806 is committed 00002000:00010000:3.0:1789760754.813312:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22478848 -> 22482944 00010000:00010000:3.0:1789760754.813328:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22478848->22482943] (req 22478848->22482943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.813342:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22478848->22482943] (req 22478848->22482943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.813355:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22478848->22482943] (req 22478848->22482943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.813363:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760754.813367:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22478848->22482943] (req 22478848->22482943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.813380:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22478848->22482943] (req 22478848->22482943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.813410:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22478848->22482943] (req 22478848->22482943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760754.813417:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fcd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22478848->22482943] (req 22478848->22482943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760754.813533:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484269184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59123us (59694us total) trans 4294972806 rc 0/0 00000100:00100000:3.0:1789760754.813550:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5517 00000100:00100000:3.0:1789760754.813559:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744038607022 time 744085326490 state 0x2 ref 0 00000100:00100000:3.0:1789760754.816461:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.816560:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484269312 00000100:00100000:1.0:1789760754.816570:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484269312 00000100:00100000:1.0:1789760754.816623:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744038607022 time 744085326490 state 0x0 ref 1 00000100:00100000:1.0:1789760754.816638:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744085586906 time 744085326490 state 0x1 ref 1 00000100:00100000:1.0:1789760754.816670:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744085586906 time 744147600786 state 0x0 ref 1 00000100:00100000:1.0:1789760754.816684:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484269312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.816734:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22482944, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760754.816743:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.816758:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:1.0:1789760754.816763:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.816770:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22482944->22487039] (req 22482944->22487039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.816784:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22482944->22487039] (req 22482944->22487039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760754.843696:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972807 is committed 00002000:00010000:2.0:1789760754.843789:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22482944 -> 22487040 00010000:00010000:2.0:1789760754.843804:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22482944->22487039] (req 22482944->22487039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.843815:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22482944->22487039] (req 22482944->22487039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.843829:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22482944->22487039] (req 22482944->22487039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.843836:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760754.843886:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22482944->22487039] (req 22482944->22487039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.843896:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22482944->22487039] (req 22482944->22487039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.843916:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22482944->22487039] (req 22482944->22487039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760754.843924:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396208fd4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22482944->22487039] (req 22482944->22487039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760754.844045:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484269312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27358us (27587us total) trans 4294972807 rc 0/0 00000100:00100000:2.0:1789760754.844056:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5518 00000100:00100000:2.0:1789760754.844112:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744085586906 time 744147600786 state 0x2 ref 0 00000100:00100000:2.0:1789760754.854543:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.854722:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484269440 00000100:00100000:1.0:1789760754.854732:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484269440 00000100:00100000:1.0:1789760754.854785:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744085586906 time 744147600786 state 0x0 ref 1 00000100:00100000:1.0:1789760754.854800:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744147861202 time 744147600786 state 0x1 ref 1 00000100:00100000:1.0:1789760754.854834:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744147861202 time 744185764659 state 0x0 ref 1 00000100:00100000:1.0:1789760754.854847:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484269440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.854895:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22487040, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760754.854904:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.854919:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:1.0:1789760754.854923:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.854931:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22487040->22491135] (req 22487040->22491135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.854943:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22487040->22491135] (req 22487040->22491135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760754.894180:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972808 is committed 00002000:00010000:1.0:1789760754.894299:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22487040 -> 22491136 00010000:00010000:1.0:1789760754.894314:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22487040->22491135] (req 22487040->22491135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.894327:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22487040->22491135] (req 22487040->22491135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.894337:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22487040->22491135] (req 22487040->22491135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.894344:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.894348:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22487040->22491135] (req 22487040->22491135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.894358:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22487040->22491135] (req 22487040->22491135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.894377:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22487040->22491135] (req 22487040->22491135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.894384:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208fdb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22487040->22491135] (req 22487040->22491135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.894471:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484269440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39622us (39934us total) trans 4294972808 rc 0/0 00000100:00100000:1.0:1789760754.894485:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5519 00000100:00100000:1.0:1789760754.894495:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744147861202 time 744185764659 state 0x2 ref 0 00000100:00100000:3.0:1789760754.897575:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760754.897884:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484269568 00000100:00100000:0.0:1789760754.897894:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484269568 00000100:00100000:0.0:1789760754.897961:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744147861202 time 744185764659 state 0x0 ref 1 00000100:00100000:0.0:1789760754.897976:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744186025075 time 744185764659 state 0x1 ref 1 00000100:00100000:0.0:1789760754.898014:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744186025075 time 744228945515 state 0x0 ref 1 00000100:00100000:0.0:1789760754.898029:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484269568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760754.898108:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22491136, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:0.0:1789760754.898117:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.898134:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:0.0:1789760754.898138:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760754.898145:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22491136->22495231] (req 22491136->22495231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.898159:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22491136->22495231] (req 22491136->22495231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760754.942906:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972809 is committed 00002000:00010000:1.0:1789760754.943053:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22491136 -> 22495232 00010000:00010000:1.0:1789760754.943197:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22491136->22495231] (req 22491136->22495231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.943211:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22491136->22495231] (req 22491136->22495231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.943235:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22491136->22495231] (req 22491136->22495231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.943243:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.943248:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22491136->22495231] (req 22491136->22495231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.943257:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22491136->22495231] (req 22491136->22495231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.943277:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22491136->22495231] (req 22491136->22495231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.943286:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396208fe2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22491136->22495231] (req 22491136->22495231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760754.943911:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484269568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45879us (46337us total) trans 4294972809 rc 0/0 00000100:00100000:0.0:1789760754.943923:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5520 00000100:00100000:0.0:1789760754.943931:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744186025075 time 744228945515 state 0x2 ref 0 00000100:00100000:3.0:1789760754.947875:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760754.948730:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484269696 00000100:00100000:0.0:1789760754.948749:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484269696 00000100:00100000:0.0:1789760754.948849:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744186025075 time 744228945515 state 0x0 ref 1 00000100:00100000:0.0:1789760754.948869:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744229205931 time 744228945515 state 0x1 ref 1 00000100:00100000:0.0:1789760754.948906:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744229205931 time 744279837290 state 0x0 ref 1 00000100:00100000:0.0:1789760754.948924:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484269696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760754.948995:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22495232, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:0.0:1789760754.949016:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.949038:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:0.0:1789760754.949044:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760754.949052:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22495232->22499327] (req 22495232->22499327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760754.949282:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22495232->22499327] (req 22495232->22499327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760754.990825:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972810 is committed 00002000:00010000:1.0:1789760754.990934:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22495232 -> 22499328 00010000:00010000:1.0:1789760754.990954:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22495232->22499327] (req 22495232->22499327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.990966:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22495232->22499327] (req 22495232->22499327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.990975:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22495232->22499327] (req 22495232->22499327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.990985:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760754.990990:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22495232->22499327] (req 22495232->22499327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.991007:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22495232->22499327] (req 22495232->22499327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.991029:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22495232->22499327] (req 22495232->22499327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.991036:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396208fe9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22495232->22499327] (req 22495232->22499327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760754.991350:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484269696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42426us (43486us total) trans 4294972810 rc 0/0 00000100:00100000:1.0:1789760754.991364:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5521 00000100:00100000:1.0:1789760754.991374:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744229205931 time 744279837290 state 0x2 ref 0 00000100:00100000:2.0:1789760754.994039:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760754.994204:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484269824 00000100:00100000:1.0:1789760754.994223:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484269824 00000100:00100000:1.0:1789760754.994286:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744229205931 time 744279837290 state 0x0 ref 1 00000100:00100000:1.0:1789760754.994302:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744280097706 time 744279837290 state 0x1 ref 1 00000100:00100000:1.0:1789760754.994376:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744280097706 time 744325298947 state 0x0 ref 1 00000100:00100000:1.0:1789760754.994389:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484269824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760754.994437:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22499328, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760754.994446:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.994459:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:1.0:1789760754.994463:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760754.994470:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22499328->22503423] (req 22499328->22503423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760754.994483:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22499328->22503423] (req 22499328->22503423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760755.026901:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972811 is committed 00002000:00010000:1.0:1789760755.026987:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22499328 -> 22503424 00010000:00010000:1.0:1789760755.027248:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22499328->22503423] (req 22499328->22503423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.027262:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22499328->22503423] (req 22499328->22503423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.027272:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22499328->22503423] (req 22499328->22503423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.027279:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760755.027284:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22499328->22503423] (req 22499328->22503423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.027293:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22499328->22503423] (req 22499328->22503423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.027312:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22499328->22503423] (req 22499328->22503423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.027321:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22499328->22503423] (req 22499328->22503423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760755.027454:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484269824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33062us (33420us total) trans 4294972811 rc 0/0 00000100:00100000:1.0:1789760755.027466:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5522 00000100:00100000:1.0:1789760755.027479:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744280097706 time 744325298947 state 0x2 ref 0 00000100:00100000:2.0:1789760755.033350:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760755.033483:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484269952 00000100:00100000:1.0:1789760755.033493:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484269952 00000100:00100000:1.0:1789760755.033542:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744280097706 time 744325298947 state 0x0 ref 1 00000100:00100000:1.0:1789760755.033553:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744325559363 time 744325298947 state 0x1 ref 1 00000100:00100000:1.0:1789760755.033597:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744325559363 time 744364527783 state 0x0 ref 1 00000100:00100000:1.0:1789760755.033610:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484269952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760755.033664:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22503424, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760755.033674:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.033693:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:1.0:1789760755.033698:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760755.033704:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22503424->22507519] (req 22503424->22507519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.033717:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22503424->22507519] (req 22503424->22507519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760755.069844:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972812 is committed 00002000:00010000:1.0:1789760755.069930:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22503424 -> 22507520 00010000:00010000:1.0:1789760755.069944:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22503424->22507519] (req 22503424->22507519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.069954:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22503424->22507519] (req 22503424->22507519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.069964:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22503424->22507519] (req 22503424->22507519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.069969:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760755.069972:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22503424->22507519] (req 22503424->22507519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.069977:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22503424->22507519] (req 22503424->22507519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.069992:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22503424->22507519] (req 22503424->22507519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.069998:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396208ff7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22503424->22507519] (req 22503424->22507519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760755.070154:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484269952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36541us (36807us total) trans 4294972812 rc 0/0 00000100:00100000:1.0:1789760755.070164:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5523 00000100:00100000:1.0:1789760755.070174:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744325559363 time 744364527783 state 0x2 ref 0 00000100:00100000:3.0:1789760755.074241:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760755.076464:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484270080 00000100:00100000:0.0:1789760755.076478:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484270080 00000100:00100000:0.0:1789760755.076541:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744325559363 time 744364527783 state 0x0 ref 1 00000100:00100000:0.0:1789760755.076553:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744364788199 time 744364527783 state 0x1 ref 1 00000100:00100000:0.0:1789760755.076579:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744364788199 time 744407510424 state 0x0 ref 1 00000100:00100000:0.0:1789760755.076593:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484270080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760755.076642:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22507520, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:0.0:1789760755.076652:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.076667:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:0.0:1789760755.076670:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760755.076675:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22507520->22511615] (req 22507520->22511615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.076685:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22507520->22511615] (req 22507520->22511615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760755.097605:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972813 is committed 00002000:00010000:0.0:1789760755.097764:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22507520 -> 22511616 00010000:00010000:0.0:1789760755.097781:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22507520->22511615] (req 22507520->22511615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.097794:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22507520->22511615] (req 22507520->22511615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.097803:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22507520->22511615] (req 22507520->22511615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.097810:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760755.097815:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22507520->22511615] (req 22507520->22511615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.097824:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22507520->22511615] (req 22507520->22511615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.097844:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22507520->22511615] (req 22507520->22511615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.097853:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396208ffe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22507520->22511615] (req 22507520->22511615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760755.097948:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484270080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 21353us (23711us total) trans 4294972813 rc 0/0 00000100:00100000:0.0:1789760755.097959:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5524 00000100:00100000:0.0:1789760755.097970:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744364788199 time 744407510424 state 0x2 ref 0 00000100:00100000:2.0:1789760755.104240:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760755.105833:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484270208 00000100:00100000:0.0:1789760755.105846:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484270208 00000100:00100000:0.0:1789760755.105912:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744364788199 time 744407510424 state 0x0 ref 1 00000100:00100000:0.0:1789760755.105927:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744407770840 time 744407510424 state 0x1 ref 1 00000100:00100000:0.0:1789760755.105963:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744407770840 time 744436893701 state 0x0 ref 1 00000100:00100000:0.0:1789760755.105977:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484270208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760755.106204:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22511616, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:0.0:1789760755.106224:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.106367:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:0.0:1789760755.106373:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760755.106380:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22511616->22515711] (req 22511616->22515711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.106395:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22511616->22515711] (req 22511616->22515711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760755.147876:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972814 is committed 00002000:00010000:1.0:1789760755.147958:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22511616 -> 22515712 00010000:00010000:1.0:1789760755.147971:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22511616->22515711] (req 22511616->22515711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.147983:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22511616->22515711] (req 22511616->22515711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.147989:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22511616->22515711] (req 22511616->22515711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.147993:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760755.148005:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22511616->22515711] (req 22511616->22515711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.148011:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22511616->22515711] (req 22511616->22515711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.148026:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22511616->22515711] (req 22511616->22515711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.148031:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209005 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22511616->22515711] (req 22511616->22515711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760755.148376:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484270208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42396us (44139us total) trans 4294972814 rc 0/0 00000100:00100000:1.0:1789760755.148386:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5525 00000100:00100000:1.0:1789760755.148396:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744407770840 time 744436893701 state 0x2 ref 0 00000100:00100000:2.0:1789760755.152577:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760755.152773:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484270336 00000100:00100000:1.0:1789760755.152782:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484270336 00000100:00100000:1.0:1789760755.152842:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744407770840 time 744436893701 state 0x0 ref 1 00000100:00100000:1.0:1789760755.152859:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744437154117 time 744436893701 state 0x1 ref 1 00000100:00100000:1.0:1789760755.152935:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744437154117 time 744483865181 state 0x0 ref 1 00000100:00100000:1.0:1789760755.152950:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484270336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760755.153017:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22515712, mtime: 0x6aad94f2, atime: 0x0, ctime: 0x6aad94f2, blocks: 0xb007 00010000:00010000:1.0:1789760755.153027:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.153044:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:1.0:1789760755.153049:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760755.153056:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22515712->22519807] (req 22515712->22519807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.153109:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22515712->22519807] (req 22515712->22519807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760755.193437:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972815 is committed 00002000:00010000:1.0:1789760755.193626:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22515712 -> 22519808 00002000:00010000:1.0:1789760755.193631:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760754 -> 1789760755 00002000:00010000:1.0:1789760755.193633:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760754 -> 1789760755 00010000:00010000:1.0:1789760755.193646:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22515712->22519807] (req 22515712->22519807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.193660:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22515712->22519807] (req 22515712->22519807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.193670:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22515712->22519807] (req 22515712->22519807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.193677:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760755.193681:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22515712->22519807] (req 22515712->22519807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.193690:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22515712->22519807] (req 22515712->22519807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.193710:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22515712->22519807] (req 22515712->22519807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.193719:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620900c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22515712->22519807] (req 22515712->22519807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760755.193816:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484270336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40864us (41243us total) trans 4294972815 rc 0/0 00000100:00100000:1.0:1789760755.193827:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5526 00000100:00100000:1.0:1789760755.193838:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744437154117 time 744483865181 state 0x2 ref 0 00000100:00100000:3.0:1789760755.199502:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760755.200471:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484270464 00000100:00100000:1.0:1789760755.200482:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484270464 00000100:00100000:1.0:1789760755.200538:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744437154117 time 744483865181 state 0x0 ref 1 00000100:00100000:1.0:1789760755.200553:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744484125597 time 744483865181 state 0x1 ref 1 00000100:00100000:1.0:1789760755.200677:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744484125597 time 744531606631 state 0x0 ref 1 00000100:00100000:1.0:1789760755.200691:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abd180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484270464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760755.200740:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22519808, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:1.0:1789760755.200749:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.200765:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:1.0:1789760755.200770:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760755.200776:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22519808->22523903] (req 22519808->22523903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.200791:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22519808->22523903] (req 22519808->22523903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760755.246205:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972816 is committed 00002000:00010000:3.0:1789760755.246358:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22519808 -> 22523904 00010000:00010000:3.0:1789760755.246372:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22519808->22523903] (req 22519808->22523903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.246385:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22519808->22523903] (req 22519808->22523903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.246394:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22519808->22523903] (req 22519808->22523903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.246401:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760755.246405:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22519808->22523903] (req 22519808->22523903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.246413:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22519808->22523903] (req 22519808->22523903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.246431:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22519808->22523903] (req 22519808->22523903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.246438:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209013 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22519808->22523903] (req 22519808->22523903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760755.246535:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abd180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484270464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45842us (47039us total) trans 4294972816 rc 0/0 00000100:00100000:3.0:1789760755.246545:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5527 00000100:00100000:3.0:1789760755.246555:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744484125597 time 744531606631 state 0x2 ref 0 00000100:00100000:3.0:1789760755.252991:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760755.254106:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484270592 00000100:00100000:0.0:1789760755.254116:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484270592 00000100:00100000:0.0:1789760755.254258:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744484125597 time 744531606631 state 0x0 ref 1 00000100:00100000:0.0:1789760755.254273:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744531867047 time 744531606631 state 0x1 ref 1 00000100:00100000:0.0:1789760755.254361:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744531867047 time 744585290928 state 0x0 ref 1 00000100:00100000:0.0:1789760755.254377:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484270592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760755.254429:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22523904, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:0.0:1789760755.254438:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.254458:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:0.0:1789760755.254463:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760755.254469:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22523904->22527999] (req 22523904->22527999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.254483:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22523904->22527999] (req 22523904->22527999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760755.328922:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972817 is committed 00002000:00010000:3.0:1789760755.329178:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22523904 -> 22528000 00010000:00010000:3.0:1789760755.329198:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22523904->22527999] (req 22523904->22527999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.329224:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22523904->22527999] (req 22523904->22527999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.329235:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22523904->22527999] (req 22523904->22527999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.329244:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760755.329249:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22523904->22527999] (req 22523904->22527999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.329260:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22523904->22527999] (req 22523904->22527999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.329285:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22523904->22527999] (req 22523904->22527999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.329293:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620901a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22523904->22527999] (req 22523904->22527999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760755.329421:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484270592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 75041us (76430us total) trans 4294972817 rc 0/0 00000100:00100000:3.0:1789760755.329433:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5528 00000100:00100000:3.0:1789760755.329445:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744531867047 time 744585290928 state 0x2 ref 0 00000100:00100000:2.0:1789760755.333368:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760755.333465:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484270720 00000100:00100000:1.0:1789760755.333478:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484270720 00000100:00100000:1.0:1789760755.333553:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744531867047 time 744585290928 state 0x0 ref 1 00000100:00100000:1.0:1789760755.333572:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744585551344 time 744585290928 state 0x1 ref 1 00000100:00100000:1.0:1789760755.333610:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744585551344 time 744664541353 state 0x0 ref 1 00000100:00100000:1.0:1789760755.333624:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484270720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760755.333687:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22528000, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:1.0:1789760755.333699:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.333720:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:1.0:1789760755.333725:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760755.333733:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22528000->22532095] (req 22528000->22532095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.333748:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22528000->22532095] (req 22528000->22532095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760755.381940:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972818 is committed 00002000:00010000:3.0:1789760755.382206:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22528000 -> 22532096 00010000:00010000:3.0:1789760755.382238:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22528000->22532095] (req 22528000->22532095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.382255:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22528000->22532095] (req 22528000->22532095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.382266:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22528000->22532095] (req 22528000->22532095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.382275:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760755.382281:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22528000->22532095] (req 22528000->22532095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.382293:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22528000->22532095] (req 22528000->22532095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.382319:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22528000->22532095] (req 22528000->22532095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.382327:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209021 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22528000->22532095] (req 22528000->22532095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760755.382458:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484270720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48828us (49098us total) trans 4294972818 rc 0/0 00000100:00100000:3.0:1789760755.382471:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5529 00000100:00100000:3.0:1789760755.382486:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744585551344 time 744664541353 state 0x2 ref 0 00000100:00100000:3.0:1789760755.385360:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760755.385490:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484270848 00000100:00100000:1.0:1789760755.385505:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484270848 00000100:00100000:1.0:1789760755.385580:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744585551344 time 744664541353 state 0x0 ref 1 00000100:00100000:1.0:1789760755.385600:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744664801769 time 744664541353 state 0x1 ref 1 00000100:00100000:1.0:1789760755.385665:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744664801769 time 744716594673 state 0x0 ref 1 00000100:00100000:1.0:1789760755.385680:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484270848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760755.385749:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22532096, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:1.0:1789760755.385760:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.385781:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:1.0:1789760755.385787:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760755.385795:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22532096->22536191] (req 22532096->22536191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.385814:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22532096->22536191] (req 22532096->22536191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760755.413223:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22532096 -> 22536192 00010000:00010000:1.0:1789760755.413252:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22532096->22536191] (req 22532096->22536191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.413268:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22532096->22536191] (req 22532096->22536191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.413287:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22532096->22536191] (req 22532096->22536191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.413295:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760755.413301:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22532096->22536191] (req 22532096->22536191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.413312:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22532096->22536191] (req 22532096->22536191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.413337:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22532096->22536191] (req 22532096->22536191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.413345:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209028 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22532096->22536191] (req 22532096->22536191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760755.413471:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484270848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27787us (28118us total) trans 4294972819 rc 0/0 00000100:00100000:1.0:1789760755.413484:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5530 00000100:00100000:1.0:1789760755.413496:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744664801769 time 744716594673 state 0x2 ref 0 00000001:00080000:0.0:1789760755.413602:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972819 is committed 00000100:00100000:1.0:1789760755.417032:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760755.417175:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484270976 00000100:00100000:3.0:1789760755.417190:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484270976 00000100:00100000:3.0:1789760755.417277:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744664801769 time 744716594673 state 0x0 ref 1 00000100:00100000:3.0:1789760755.417295:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744716855089 time 744716594673 state 0x1 ref 1 00000100:00100000:3.0:1789760755.417341:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744716855089 time 744748271061 state 0x0 ref 1 00000100:00100000:3.0:1789760755.417357:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484270976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760755.417561:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22536192, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:3.0:1789760755.417574:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.417594:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:3.0:1789760755.417600:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760755.417608:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22536192->22540287] (req 22536192->22540287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.417625:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22536192->22540287] (req 22536192->22540287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760755.455499:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972820 is committed 00002000:00010000:1.0:1789760755.455601:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22536192 -> 22540288 00010000:00010000:1.0:1789760755.455619:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22536192->22540287] (req 22536192->22540287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.455630:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22536192->22540287] (req 22536192->22540287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.455638:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22536192->22540287] (req 22536192->22540287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.455643:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760755.455648:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22536192->22540287] (req 22536192->22540287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.455656:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22536192->22540287] (req 22536192->22540287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.455675:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22536192->22540287] (req 22536192->22540287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.455685:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620902f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22536192->22540287] (req 22536192->22540287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760755.455816:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484270976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38457us (38793us total) trans 4294972820 rc 0/0 00000100:00100000:1.0:1789760755.455824:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5531 00000100:00100000:1.0:1789760755.455834:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744716855089 time 744748271061 state 0x2 ref 0 00000100:00100000:2.0:1789760755.458180:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760755.458411:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484271104 00000100:00100000:3.0:1789760755.458424:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484271104 00000100:00100000:3.0:1789760755.458497:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744716855089 time 744748271061 state 0x0 ref 1 00000100:00100000:3.0:1789760755.458513:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744748531477 time 744748271061 state 0x1 ref 1 00000100:00100000:3.0:1789760755.458540:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744748531477 time 744789470509 state 0x0 ref 1 00000100:00100000:3.0:1789760755.458554:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484271104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760755.458628:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22540288, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:3.0:1789760755.458638:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.458659:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:3.0:1789760755.458664:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760755.458673:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22540288->22544383] (req 22540288->22544383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.458689:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22540288->22544383] (req 22540288->22544383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760755.494945:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972821 is committed 00002000:00010000:3.0:1789760755.495187:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22540288 -> 22544384 00010000:00010000:3.0:1789760755.495208:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22540288->22544383] (req 22540288->22544383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.495230:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22540288->22544383] (req 22540288->22544383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.495241:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22540288->22544383] (req 22540288->22544383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.495250:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760755.495256:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22540288->22544383] (req 22540288->22544383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.495266:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22540288->22544383] (req 22540288->22544383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.495290:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22540288->22544383] (req 22540288->22544383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.495362:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209036 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22540288->22544383] (req 22540288->22544383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760755.495598:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484271104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37038us (37423us total) trans 4294972821 rc 0/0 00000100:00100000:3.0:1789760755.495610:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5532 00000100:00100000:3.0:1789760755.495622:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744748531477 time 744789470509 state 0x2 ref 0 00000100:00100000:3.0:1789760755.500048:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760755.500283:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484271232 00000100:00100000:2.0:1789760755.500296:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484271232 00000100:00100000:2.0:1789760755.500375:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744748531477 time 744789470509 state 0x0 ref 1 00000100:00100000:2.0:1789760755.500394:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744789730925 time 744789470509 state 0x1 ref 1 00000100:00100000:2.0:1789760755.500422:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744789730925 time 744831353060 state 0x0 ref 1 00000100:00100000:2.0:1789760755.500434:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484271232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760755.500506:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22544384, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:2.0:1789760755.500517:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.500537:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:2.0:1789760755.500543:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760755.500550:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22544384->22548479] (req 22544384->22548479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.500567:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22544384->22548479] (req 22544384->22548479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760755.553321:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22544384 -> 22548480 00010000:00010000:3.0:1789760755.553351:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22544384->22548479] (req 22544384->22548479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.553367:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22544384->22548479] (req 22544384->22548479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.553384:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22544384->22548479] (req 22544384->22548479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.553392:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760755.553398:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22544384->22548479] (req 22544384->22548479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.553408:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22544384->22548479] (req 22544384->22548479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.553433:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22544384->22548479] (req 22544384->22548479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.553441:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620903d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22544384->22548479] (req 22544384->22548479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760755.553567:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484271232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53127us (53527us total) trans 4294972822 rc 0/0 00000100:00100000:3.0:1789760755.553580:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5533 00000100:00100000:3.0:1789760755.553592:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744789730925 time 744831353060 state 0x2 ref 0 00000001:00080000:0.0:1789760755.555263:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972822 is committed 00000100:00100000:1.0:1789760755.557575:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760755.557870:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484271360 00000100:00100000:2.0:1789760755.557882:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484271360 00000100:00100000:2.0:1789760755.557961:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744789730925 time 744831353060 state 0x0 ref 1 00000100:00100000:2.0:1789760755.557981:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744831613476 time 744831353060 state 0x1 ref 1 00000100:00100000:2.0:1789760755.558014:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744831613476 time 744888944418 state 0x0 ref 1 00000100:00100000:2.0:1789760755.558029:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484271360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760755.558457:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22548480, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:2.0:1789760755.558469:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.558489:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:2.0:1789760755.558495:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760755.558504:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22548480->22552575] (req 22548480->22552575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.558522:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22548480->22552575] (req 22548480->22552575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.596895:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760755.605544:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972823 is committed 00002000:00010000:2.0:1789760755.605747:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22548480 -> 22552576 00010000:00010000:2.0:1789760755.605763:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22548480->22552575] (req 22548480->22552575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.605783:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22548480->22552575] (req 22548480->22552575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.605793:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22548480->22552575] (req 22548480->22552575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.605801:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760755.605806:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22548480->22552575] (req 22548480->22552575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.605816:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22548480->22552575] (req 22548480->22552575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.605839:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22548480->22552575] (req 22548480->22552575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.605847:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209044 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22548480->22552575] (req 22548480->22552575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760755.605941:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484271360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47910us (48375us total) trans 4294972823 rc 0/0 00000100:00100000:2.0:1789760755.605951:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5534 00000100:00100000:2.0:1789760755.605962:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744831613476 time 744888944418 state 0x2 ref 0 00000100:00100000:1.0:1789760755.609377:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760755.609542:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484271488 00000100:00100000:2.0:1789760755.609552:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484271488 00000100:00100000:2.0:1789760755.609673:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744831613476 time 744888944418 state 0x0 ref 1 00000100:00100000:2.0:1789760755.609688:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744889204834 time 744888944418 state 0x1 ref 1 00000100:00100000:2.0:1789760755.609764:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744889204834 time 744940694697 state 0x0 ref 1 00000100:00100000:2.0:1789760755.609778:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b82a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484271488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760755.609829:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22552576, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:2.0:1789760755.609839:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.609855:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:2.0:1789760755.609859:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760755.609865:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22552576->22556671] (req 22552576->22556671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.609878:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22552576->22556671] (req 22552576->22556671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760755.647815:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972824 is committed 00002000:00010000:1.0:1789760755.647927:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22552576 -> 22556672 00010000:00010000:1.0:1789760755.647945:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22552576->22556671] (req 22552576->22556671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.647958:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22552576->22556671] (req 22552576->22556671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.647967:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22552576->22556671] (req 22552576->22556671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.647976:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760755.647981:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22552576->22556671] (req 22552576->22556671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.647996:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22552576->22556671] (req 22552576->22556671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.648017:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22552576->22556671] (req 22552576->22556671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.648025:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620904b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22552576->22556671] (req 22552576->22556671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760755.648330:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b82a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484271488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38549us (38957us total) trans 4294972824 rc 0/0 00000100:00100000:1.0:1789760755.648343:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5535 00000100:00100000:1.0:1789760755.648353:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744889204834 time 744940694697 state 0x2 ref 0 00000100:00100000:2.0:1789760755.652899:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760755.652986:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484271616 00000100:00100000:3.0:1789760755.652996:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484271616 00000100:00100000:3.0:1789760755.653049:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744889204834 time 744940694697 state 0x0 ref 1 00000100:00100000:3.0:1789760755.653093:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744940955113 time 744940694697 state 0x1 ref 1 00000100:00100000:3.0:1789760755.653127:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744940955113 time 744984058349 state 0x0 ref 1 00000100:00100000:3.0:1789760755.653139:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ad880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484271616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760755.653188:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22556672, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:3.0:1789760755.653196:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.653227:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=744s v=(0 0) 00010000:00010000:3.0:1789760755.653232:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760755.653238:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22556672->22560767] (req 22556672->22560767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.653253:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22556672->22560767] (req 22556672->22560767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760755.697698:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972825 is committed 00002000:00010000:3.0:1789760755.697842:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22556672 -> 22560768 00010000:00010000:3.0:1789760755.697857:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22556672->22560767] (req 22556672->22560767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.697870:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22556672->22560767] (req 22556672->22560767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.697879:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22556672->22560767] (req 22556672->22560767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.697885:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760755.697890:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22556672->22560767] (req 22556672->22560767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.697899:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22556672->22560767] (req 22556672->22560767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.697922:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22556672->22560767] (req 22556672->22560767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.697930:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209052 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22556672->22560767] (req 22556672->22560767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760755.698032:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ad880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484271616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44890us (45137us total) trans 4294972825 rc 0/0 00000100:00100000:3.0:1789760755.698042:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5536 00000100:00100000:3.0:1789760755.698051:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744940955113 time 744984058349 state 0x2 ref 0 00000100:00100000:0.0:1789760755.701701:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760755.701777:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484271744 00000100:00100000:3.0:1789760755.701788:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484271744 00000100:00100000:3.0:1789760755.701843:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744940955113 time 744984058349 state 0x0 ref 1 00000100:00100000:3.0:1789760755.701858:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744984318765 time 744984058349 state 0x1 ref 1 00000100:00100000:3.0:1789760755.701891:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744984318765 time 745032821847 state 0x0 ref 1 00000100:00100000:3.0:1789760755.701904:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484271744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760755.701950:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22560768, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:3.0:1789760755.701959:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.701975:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:3.0:1789760755.701979:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760755.701985:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22560768->22564863] (req 22560768->22564863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.701997:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22560768->22564863] (req 22560768->22564863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760755.766983:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972826 is committed 00002000:00010000:2.0:1789760755.767290:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22560768 -> 22564864 00010000:00010000:2.0:1789760755.767306:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22560768->22564863] (req 22560768->22564863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.767319:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22560768->22564863] (req 22560768->22564863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.767334:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22560768->22564863] (req 22560768->22564863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.767342:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760755.767347:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22560768->22564863] (req 22560768->22564863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.767357:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22560768->22564863] (req 22560768->22564863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.767379:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22560768->22564863] (req 22560768->22564863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.767387:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209059 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22560768->22564863] (req 22560768->22564863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760755.767488:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484271744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65581us (65792us total) trans 4294972826 rc 0/0 00000100:00100000:2.0:1789760755.767499:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5537 00000100:00100000:2.0:1789760755.767509:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744984318765 time 745032821847 state 0x2 ref 0 00000100:00100000:1.0:1789760755.769890:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760755.769977:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484271872 00000100:00100000:0.0:1789760755.769988:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484271872 00000100:00100000:0.0:1789760755.770058:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 744984318765 time 745032821847 state 0x0 ref 1 00000100:00100000:0.0:1789760755.770105:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745033082263 time 745032821847 state 0x1 ref 1 00000100:00100000:0.0:1789760755.770133:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745033082263 time 745101063910 state 0x0 ref 1 00000100:00100000:0.0:1789760755.770146:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484271872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760755.770198:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22564864, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:0.0:1789760755.770219:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.770235:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:0.0:1789760755.770239:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760755.770245:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22564864->22568959] (req 22564864->22568959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760755.770259:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22564864->22568959] (req 22564864->22568959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760755.799711:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972827 is committed 00002000:00010000:1.0:1789760755.799796:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22564864 -> 22568960 00010000:00010000:1.0:1789760755.799810:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22564864->22568959] (req 22564864->22568959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.799823:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22564864->22568959] (req 22564864->22568959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.799832:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22564864->22568959] (req 22564864->22568959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.799839:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760755.799844:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22564864->22568959] (req 22564864->22568959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.799854:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22564864->22568959] (req 22564864->22568959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.799873:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22564864->22568959] (req 22564864->22568959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.799880:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209060 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22564864->22568959] (req 22564864->22568959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760755.800016:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484271872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29868us (30130us total) trans 4294972827 rc 0/0 00000100:00100000:1.0:1789760755.800026:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5538 00000100:00100000:1.0:1789760755.800036:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745033082263 time 745101063910 state 0x2 ref 0 00000100:00100000:0.0:1789760755.803725:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760755.803809:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484272000 00000100:00100000:1.0:1789760755.803819:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484272000 00000100:00100000:1.0:1789760755.803880:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745033082263 time 745101063910 state 0x0 ref 1 00000100:00100000:1.0:1789760755.803895:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745101324326 time 745101063910 state 0x1 ref 1 00000100:00100000:1.0:1789760755.803928:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745101324326 time 745134858789 state 0x0 ref 1 00000100:00100000:1.0:1789760755.803941:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484272000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760755.803990:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22568960, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:1.0:1789760755.804006:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.804022:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:1.0:1789760755.804026:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760755.804032:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22568960->22573055] (req 22568960->22573055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.804046:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22568960->22573055] (req 22568960->22573055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760755.865282:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972828 is committed 00002000:00010000:1.0:1789760755.865471:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22568960 -> 22573056 00010000:00010000:1.0:1789760755.865487:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22568960->22573055] (req 22568960->22573055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.865500:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22568960->22573055] (req 22568960->22573055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.865508:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22568960->22573055] (req 22568960->22573055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.865515:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760755.865520:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22568960->22573055] (req 22568960->22573055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.865529:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22568960->22573055] (req 22568960->22573055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.865553:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22568960->22573055] (req 22568960->22573055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.865561:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209067 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22568960->22573055] (req 22568960->22573055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760755.865657:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484272000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61714us (61935us total) trans 4294972828 rc 0/0 00000100:00100000:1.0:1789760755.865668:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5539 00000100:00100000:1.0:1789760755.865678:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745101324326 time 745134858789 state 0x2 ref 0 00000100:00100000:2.0:1789760755.874698:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760755.874807:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484272128 00000100:00100000:3.0:1789760755.874825:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484272128 00000100:00100000:3.0:1789760755.874906:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745101324326 time 745134858789 state 0x0 ref 1 00000100:00100000:3.0:1789760755.874926:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745135119205 time 745134858789 state 0x1 ref 1 00000100:00100000:3.0:1789760755.875043:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745135119205 time 745205972106 state 0x0 ref 1 00000100:00100000:3.0:1789760755.875060:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484272128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760755.875168:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22573056, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:3.0:1789760755.875179:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.875214:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:3.0:1789760755.875220:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760755.875229:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22573056->22577151] (req 22573056->22577151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.875248:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22573056->22577151] (req 22573056->22577151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760755.948864:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22573056 -> 22577152 00010000:00010000:3.0:1789760755.948893:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22573056->22577151] (req 22573056->22577151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.948907:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22573056->22577151] (req 22573056->22577151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.948916:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22573056->22577151] (req 22573056->22577151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.948924:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760755.948929:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22573056->22577151] (req 22573056->22577151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.948939:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22573056->22577151] (req 22573056->22577151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.948964:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22573056->22577151] (req 22573056->22577151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.948973:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620906e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22573056->22577151] (req 22573056->22577151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760755.949156:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484272128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 74092us (74460us total) trans 4294972829 rc 0/0 00000100:00100000:3.0:1789760755.949180:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5540 00000100:00100000:3.0:1789760755.949193:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745135119205 time 745205972106 state 0x2 ref 0 00000001:00080000:2.0:1789760755.949881:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972829 is committed 00000100:00100000:2.0:1789760755.955429:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760755.955892:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484272256 00000100:00100000:3.0:1789760755.955908:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484272256 00000100:00100000:3.0:1789760755.955990:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745135119205 time 745205972106 state 0x0 ref 1 00000100:00100000:3.0:1789760755.956012:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745206232522 time 745205972106 state 0x1 ref 1 00000100:00100000:3.0:1789760755.956137:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745206232522 time 745287065703 state 0x0 ref 1 00000100:00100000:3.0:1789760755.956154:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484272256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760755.956235:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22577152, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:3.0:1789760755.956246:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.956265:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:3.0:1789760755.956271:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760755.956279:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22577152->22581247] (req 22577152->22581247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760755.956306:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22577152->22581247] (req 22577152->22581247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760755.986509:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22577152 -> 22581248 00010000:00010000:2.0:1789760755.986538:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22577152->22581247] (req 22577152->22581247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.986555:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22577152->22581247] (req 22577152->22581247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.986566:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22577152->22581247] (req 22577152->22581247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.986574:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760755.986580:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22577152->22581247] (req 22577152->22581247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.986590:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22577152->22581247] (req 22577152->22581247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.986617:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22577152->22581247] (req 22577152->22581247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760755.986625:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209075 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22577152->22581247] (req 22577152->22581247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760755.986745:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12aea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484272256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30588us (31324us total) trans 4294972830 rc 0/0 00000100:00100000:2.0:1789760755.986759:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5541 00000100:00100000:2.0:1789760755.986770:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745206232522 time 745287065703 state 0x2 ref 0 00000001:00080000:2.0:1789760755.986821:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972830 is committed 00000100:00100000:3.0:1789760755.992675:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760755.992993:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484272384 00000100:00100000:1.0:1789760755.993132:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484272384 00000100:00100000:1.0:1789760755.993219:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745206232522 time 745287065703 state 0x0 ref 1 00000100:00100000:1.0:1789760755.993242:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745287326119 time 745287065703 state 0x1 ref 1 00000100:00100000:1.0:1789760755.993278:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745287326119 time 745324208070 state 0x0 ref 1 00000100:00100000:1.0:1789760755.993292:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484272384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760755.993371:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22581248, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:1.0:1789760755.993382:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.993401:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:1.0:1789760755.993407:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760755.993415:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22581248->22585343] (req 22581248->22585343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760755.993431:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22581248->22585343] (req 22581248->22585343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760756.031953:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972831 is committed 00002000:00010000:1.0:1789760756.032220:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22581248 -> 22585344 00010000:00010000:1.0:1789760756.032240:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22581248->22585343] (req 22581248->22585343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.032257:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22581248->22585343] (req 22581248->22585343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.032268:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22581248->22585343] (req 22581248->22585343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.032276:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760756.032281:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22581248->22585343] (req 22581248->22585343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.032306:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22581248->22585343] (req 22581248->22585343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.032333:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22581248->22585343] (req 22581248->22585343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.032347:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620907c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22581248->22585343] (req 22581248->22585343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760756.032577:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484272384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39280us (39906us total) trans 4294972831 rc 0/0 00000100:00100000:1.0:1789760756.032590:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5542 00000100:00100000:1.0:1789760756.032603:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745287326119 time 745324208070 state 0x2 ref 0 00000100:00100000:0.0:1789760756.035602:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760756.035800:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484272512 00000100:00100000:1.0:1789760756.035813:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484272512 00000100:00100000:1.0:1789760756.035941:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745287326119 time 745324208070 state 0x0 ref 1 00000100:00100000:1.0:1789760756.035960:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745324468486 time 745324208070 state 0x1 ref 1 00000100:00100000:1.0:1789760756.036008:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745324468486 time 745366932806 state 0x0 ref 1 00000100:00100000:1.0:1789760756.036024:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484272512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760756.036116:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22585344, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:1.0:1789760756.036127:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.036146:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:1.0:1789760756.036150:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760756.036157:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22585344->22589439] (req 22585344->22589439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.036177:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22585344->22589439] (req 22585344->22589439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760756.037045:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:3.0:1789760756.086838:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972832 is committed 00002000:00010000:1.0:1789760756.086952:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22585344 -> 22589440 00010000:00010000:1.0:1789760756.086972:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22585344->22589439] (req 22585344->22589439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.086999:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22585344->22589439] (req 22585344->22589439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.087016:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22585344->22589439] (req 22585344->22589439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.087024:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760756.087030:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22585344->22589439] (req 22585344->22589439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.087040:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22585344->22589439] (req 22585344->22589439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.087086:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22585344->22589439] (req 22585344->22589439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.087095:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209083 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22585344->22589439] (req 22585344->22589439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760756.087228:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484272512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51200us (51633us total) trans 4294972832 rc 0/0 00000100:00100000:1.0:1789760756.087240:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5543 00000100:00100000:1.0:1789760756.087253:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745324468486 time 745366932806 state 0x2 ref 0 00000100:00100000:2.0:1789760756.090553:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760756.090656:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484272640 00000100:00100000:3.0:1789760756.090670:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484272640 00000100:00100000:3.0:1789760756.090743:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745324468486 time 745366932806 state 0x0 ref 1 00000100:00100000:3.0:1789760756.090762:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745367193222 time 745366932806 state 0x1 ref 1 00000100:00100000:3.0:1789760756.090797:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745367193222 time 745421726989 state 0x0 ref 1 00000100:00100000:3.0:1789760756.090811:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484272640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760756.090874:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22589440, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:3.0:1789760756.090885:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.090905:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:3.0:1789760756.091187:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760756.091197:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22589440->22593535] (req 22589440->22593535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.091225:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22589440->22593535] (req 22589440->22593535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760756.173946:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972833 is committed 00002000:00010000:0.0:1789760756.174298:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22589440 -> 22593536 00010000:00010000:0.0:1789760756.174319:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22589440->22593535] (req 22589440->22593535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.174336:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22589440->22593535] (req 22589440->22593535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.174352:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22589440->22593535] (req 22589440->22593535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.174361:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760756.174368:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22589440->22593535] (req 22589440->22593535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.174378:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22589440->22593535] (req 22589440->22593535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.174404:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22589440->22593535] (req 22589440->22593535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.174412:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620908a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22589440->22593535] (req 22589440->22593535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760756.174541:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484272640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 83726us (84066us total) trans 4294972833 rc 0/0 00000100:00100000:0.0:1789760756.174555:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5544 00000100:00100000:0.0:1789760756.174567:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745367193222 time 745421726989 state 0x2 ref 0 00000100:00100000:3.0:1789760756.177433:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760756.177816:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484272768 00000100:00100000:2.0:1789760756.177830:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484272768 00000100:00100000:2.0:1789760756.177905:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745367193222 time 745421726989 state 0x0 ref 1 00000100:00100000:2.0:1789760756.177925:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745421987405 time 745421726989 state 0x1 ref 1 00000100:00100000:2.0:1789760756.178429:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745421987405 time 745509356145 state 0x0 ref 1 00000100:00100000:2.0:1789760756.178448:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484272768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760756.178518:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22593536, mtime: 0x6aad94f3, atime: 0x0, ctime: 0x6aad94f3, blocks: 0xb007 00010000:00010000:2.0:1789760756.178530:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.178552:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:2.0:1789760756.178558:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760756.178566:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22593536->22597631] (req 22593536->22597631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.178585:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22593536->22597631] (req 22593536->22597631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760756.241396:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972834 is committed 00002000:00010000:2.0:1789760756.241498:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22593536 -> 22597632 00002000:00010000:2.0:1789760756.241504:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760755 -> 1789760756 00002000:00010000:2.0:1789760756.241506:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760755 -> 1789760756 00010000:00010000:2.0:1789760756.241521:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22593536->22597631] (req 22593536->22597631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.241533:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22593536->22597631] (req 22593536->22597631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.241543:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22593536->22597631] (req 22593536->22597631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.241550:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760756.241555:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22593536->22597631] (req 22593536->22597631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.241563:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22593536->22597631] (req 22593536->22597631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.241583:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22593536->22597631] (req 22593536->22597631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.241590:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209091 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22593536->22597631] (req 22593536->22597631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760756.241683:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484272768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63235us (64261us total) trans 4294972834 rc 0/0 00000100:00100000:2.0:1789760756.241693:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5545 00000100:00100000:2.0:1789760756.241704:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745421987405 time 745509356145 state 0x2 ref 0 00000100:00100000:2.0:1789760756.247368:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760756.247520:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484272896 00000100:00100000:1.0:1789760756.247531:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484272896 00000100:00100000:1.0:1789760756.247585:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745421987405 time 745509356145 state 0x0 ref 1 00000100:00100000:1.0:1789760756.247600:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745509616561 time 745509356145 state 0x1 ref 1 00000100:00100000:1.0:1789760756.247633:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745509616561 time 745578563879 state 0x0 ref 1 00000100:00100000:1.0:1789760756.247646:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484272896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760756.247697:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22597632, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:1.0:1789760756.247706:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.247724:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:1.0:1789760756.247728:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760756.247736:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22597632->22601727] (req 22597632->22601727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.247751:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22597632->22601727] (req 22597632->22601727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760756.303507:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972835 is committed 00002000:00010000:0.0:1789760756.303915:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22597632 -> 22601728 00010000:00010000:0.0:1789760756.303932:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22597632->22601727] (req 22597632->22601727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.303962:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22597632->22601727] (req 22597632->22601727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.303972:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22597632->22601727] (req 22597632->22601727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.303978:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760756.303983:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22597632->22601727] (req 22597632->22601727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.304127:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22597632->22601727] (req 22597632->22601727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.304149:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22597632->22601727] (req 22597632->22601727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.304157:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209098 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22597632->22601727] (req 22597632->22601727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760756.304311:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484272896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56661us (56949us total) trans 4294972835 rc 0/0 00000100:00100000:0.0:1789760756.304321:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5546 00000100:00100000:0.0:1789760756.304331:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745509616561 time 745578563879 state 0x2 ref 0 00000100:00100000:3.0:1789760756.307639:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760756.307710:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484273024 00000100:00100000:1.0:1789760756.307719:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484273024 00000100:00100000:1.0:1789760756.307771:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745509616561 time 745578563879 state 0x0 ref 1 00000100:00100000:1.0:1789760756.307797:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745578824295 time 745578563879 state 0x1 ref 1 00000100:00100000:1.0:1789760756.307825:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745578824295 time 745638756416 state 0x0 ref 1 00000100:00100000:1.0:1789760756.307835:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484273024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760756.307877:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22601728, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:1.0:1789760756.307886:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.307901:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:1.0:1789760756.307905:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760756.307910:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22601728->22605823] (req 22601728->22605823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760756.307922:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22601728->22605823] (req 22601728->22605823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760756.343435:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972836 is committed 00002000:00010000:2.0:1789760756.343549:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22601728 -> 22605824 00010000:00010000:2.0:1789760756.343563:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22601728->22605823] (req 22601728->22605823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.343574:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22601728->22605823] (req 22601728->22605823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.343584:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22601728->22605823] (req 22601728->22605823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.343592:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760756.343597:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22601728->22605823] (req 22601728->22605823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.343606:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22601728->22605823] (req 22601728->22605823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.343626:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22601728->22605823] (req 22601728->22605823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.343633:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620909f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22601728->22605823] (req 22601728->22605823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760756.343722:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484273024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35883us (36088us total) trans 4294972836 rc 0/0 00000100:00100000:2.0:1789760756.343733:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5547 00000100:00100000:2.0:1789760756.343742:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745578824295 time 745638756416 state 0x2 ref 0 00000100:00100000:0.0:1789760756.348232:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760756.348465:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484273152 00000100:00100000:3.0:1789760756.348475:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484273152 00000100:00100000:3.0:1789760756.348528:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745578824295 time 745638756416 state 0x0 ref 1 00000100:00100000:3.0:1789760756.348543:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745639016832 time 745638756416 state 0x1 ref 1 00000100:00100000:3.0:1789760756.348610:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745639016832 time 745679539901 state 0x0 ref 1 00000100:00100000:3.0:1789760756.348623:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484273152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760756.348669:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22605824, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:3.0:1789760756.348679:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.348694:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:3.0:1789760756.348699:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760756.348705:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22605824->22609919] (req 22605824->22609919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.348719:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22605824->22609919] (req 22605824->22609919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760756.379018:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22605824 -> 22609920 00010000:00010000:0.0:1789760756.379039:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22605824->22609919] (req 22605824->22609919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.379052:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22605824->22609919] (req 22605824->22609919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.379062:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22605824->22609919] (req 22605824->22609919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.379098:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760756.379103:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22605824->22609919] (req 22605824->22609919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.379113:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22605824->22609919] (req 22605824->22609919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.379134:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22605824->22609919] (req 22605824->22609919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.379141:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090a6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22605824->22609919] (req 22605824->22609919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760756.379242:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484273152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30616us (31015us total) trans 4294972837 rc 0/0 00000100:00100000:0.0:1789760756.379252:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5548 00000100:00100000:0.0:1789760756.379262:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745639016832 time 745679539901 state 0x2 ref 0 00000001:00080000:1.0:1789760756.379396:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972837 is committed 00000100:00100000:2.0:1789760756.382896:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760756.382969:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484273280 00000100:00100000:3.0:1789760756.382978:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484273280 00000100:00100000:3.0:1789760756.383042:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745639016832 time 745679539901 state 0x0 ref 1 00000100:00100000:3.0:1789760756.383058:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745679800317 time 745679539901 state 0x1 ref 1 00000100:00100000:3.0:1789760756.383144:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745679800317 time 745714074273 state 0x0 ref 1 00000100:00100000:3.0:1789760756.383156:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484273280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760756.383214:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22609920, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:3.0:1789760756.383224:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.383239:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:3.0:1789760756.383243:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760756.383250:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22609920->22614015] (req 22609920->22614015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.383264:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22609920->22614015] (req 22609920->22614015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760756.411279:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972838 is committed 00002000:00010000:2.0:1789760756.411605:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22609920 -> 22614016 00010000:00010000:2.0:1789760756.411622:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22609920->22614015] (req 22609920->22614015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.411634:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22609920->22614015] (req 22609920->22614015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.411643:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22609920->22614015] (req 22609920->22614015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.411651:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760756.411656:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22609920->22614015] (req 22609920->22614015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.411665:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22609920->22614015] (req 22609920->22614015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.411685:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22609920->22614015] (req 22609920->22614015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.411693:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090ad lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22609920->22614015] (req 22609920->22614015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760756.411815:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ace00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484273280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28656us (28924us total) trans 4294972838 rc 0/0 00000100:00100000:2.0:1789760756.411833:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5549 00000100:00100000:2.0:1789760756.411844:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745679800317 time 745714074273 state 0x2 ref 0 00000100:00100000:2.0:1789760756.417225:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760756.417287:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484273408 00000100:00100000:0.0:1789760756.417358:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484273408 00000100:00100000:0.0:1789760756.417418:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745679800317 time 745714074273 state 0x0 ref 1 00000100:00100000:0.0:1789760756.417433:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745714334689 time 745714074273 state 0x1 ref 1 00000100:00100000:0.0:1789760756.417502:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745714334689 time 745748431957 state 0x0 ref 1 00000100:00100000:0.0:1789760756.417516:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484273408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760756.417566:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22614016, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:0.0:1789760756.417575:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.417590:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:0.0:1789760756.417594:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760756.417600:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22614016->22618111] (req 22614016->22618111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.417614:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22614016->22618111] (req 22614016->22618111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760756.452228:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972839 is committed 00002000:00010000:2.0:1789760756.452343:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22614016 -> 22618112 00010000:00010000:2.0:1789760756.452359:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22614016->22618111] (req 22614016->22618111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.452371:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22614016->22618111] (req 22614016->22618111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.452380:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22614016->22618111] (req 22614016->22618111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.452387:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760756.452392:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22614016->22618111] (req 22614016->22618111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.452401:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22614016->22618111] (req 22614016->22618111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.452421:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22614016->22618111] (req 22614016->22618111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.452434:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962090b4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22614016->22618111] (req 22614016->22618111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760756.452536:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484273408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35019us (35316us total) trans 4294972839 rc 0/0 00000100:00100000:2.0:1789760756.452547:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5550 00000100:00100000:2.0:1789760756.452556:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745714334689 time 745748431957 state 0x2 ref 0 00000100:00100000:0.0:1789760756.456427:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760756.456825:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484273536 00000100:00100000:2.0:1789760756.456834:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484273536 00000100:00100000:2.0:1789760756.456890:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745714334689 time 745748431957 state 0x0 ref 1 00000100:00100000:2.0:1789760756.456905:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745748692373 time 745748431957 state 0x1 ref 1 00000100:00100000:2.0:1789760756.456938:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745748692373 time 745787869759 state 0x0 ref 1 00000100:00100000:2.0:1789760756.456951:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484273536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760756.457004:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22618112, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:2.0:1789760756.457012:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.457027:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:2.0:1789760756.457031:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760756.457186:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22618112->22622207] (req 22618112->22622207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.457421:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22618112->22622207] (req 22618112->22622207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760756.563732:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972840 is committed 00002000:00010000:3.0:1789760756.564224:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22618112 -> 22622208 00010000:00010000:3.0:1789760756.564304:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22618112->22622207] (req 22618112->22622207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.564318:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22618112->22622207] (req 22618112->22622207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.564326:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22618112->22622207] (req 22618112->22622207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.564334:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760756.564339:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22618112->22622207] (req 22618112->22622207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.564348:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22618112->22622207] (req 22618112->22622207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.564366:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22618112->22622207] (req 22618112->22622207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.564373:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962090bb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22618112->22622207] (req 22618112->22622207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760756.564466:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484273536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 107512us (108046us total) trans 4294972840 rc 0/0 00000100:00100000:3.0:1789760756.564477:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5551 00000100:00100000:3.0:1789760756.564486:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745748692373 time 745787869759 state 0x2 ref 0 00000100:00100000:0.0:1789760756.568017:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760756.568188:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484273664 00000100:00100000:3.0:1789760756.568210:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484273664 00000100:00100000:3.0:1789760756.568265:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745748692373 time 745787869759 state 0x0 ref 1 00000100:00100000:3.0:1789760756.568280:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745788130175 time 745787869759 state 0x1 ref 1 00000100:00100000:3.0:1789760756.568310:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745788130175 time 745899240560 state 0x0 ref 1 00000100:00100000:3.0:1789760756.568321:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484273664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760756.568369:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22622208, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:3.0:1789760756.568379:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.568394:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:3.0:1789760756.568399:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760756.568406:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22622208->22626303] (req 22622208->22626303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.568420:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22622208->22626303] (req 22622208->22626303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760756.595701:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972841 is committed 00002000:00010000:3.0:1789760756.595842:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22622208 -> 22626304 00010000:00010000:3.0:1789760756.595857:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22622208->22626303] (req 22622208->22626303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.595869:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22622208->22626303] (req 22622208->22626303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.595878:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22622208->22626303] (req 22622208->22626303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.595885:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760756.595889:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22622208->22626303] (req 22622208->22626303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.595898:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22622208->22626303] (req 22622208->22626303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.595924:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22622208->22626303] (req 22622208->22626303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.595933:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22622208->22626303] (req 22622208->22626303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760756.596035:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484273664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27710us (28022us total) trans 4294972841 rc 0/0 00000100:00100000:3.0:1789760756.596046:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5552 00000100:00100000:3.0:1789760756.596055:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745788130175 time 745899240560 state 0x2 ref 0 00000100:00100000:2.0:1789760756.601315:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760756.601437:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484273792 00000100:00100000:3.0:1789760756.601447:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484273792 00000100:00100000:3.0:1789760756.601500:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745788130175 time 745899240560 state 0x0 ref 1 00000100:00100000:3.0:1789760756.601516:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745899500976 time 745899240560 state 0x1 ref 1 00000100:00100000:3.0:1789760756.601548:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745899500976 time 745932478436 state 0x0 ref 1 00000100:00100000:3.0:1789760756.601560:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12af800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484273792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760756.601608:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22626304, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:3.0:1789760756.601617:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.601633:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:3.0:1789760756.601637:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760756.601644:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22626304->22630399] (req 22626304->22630399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.601658:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22626304->22630399] (req 22626304->22630399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.620322:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:0.0:1789760756.630768:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22626304 -> 22630400 00010000:00010000:0.0:1789760756.630786:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22626304->22630399] (req 22626304->22630399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.630801:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22626304->22630399] (req 22626304->22630399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.630810:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22626304->22630399] (req 22626304->22630399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.630817:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760756.630821:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22626304->22630399] (req 22626304->22630399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.630831:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22626304->22630399] (req 22626304->22630399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.630853:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22626304->22630399] (req 22626304->22630399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.630860:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962090c9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22626304->22630399] (req 22626304->22630399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760756.630954:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12af800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484273792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29391us (29644us total) trans 4294972842 rc 0/0 00000100:00100000:0.0:1789760756.630965:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5553 00000100:00100000:0.0:1789760756.630974:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745899500976 time 745932478436 state 0x2 ref 0 00000001:00080000:0.0:1789760756.631020:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972842 is committed 00000100:00100000:0.0:1789760756.635459:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760756.635568:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484273920 00000100:00100000:2.0:1789760756.635577:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484273920 00000100:00100000:2.0:1789760756.635632:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745899500976 time 745932478436 state 0x0 ref 1 00000100:00100000:2.0:1789760756.635647:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745932738852 time 745932478436 state 0x1 ref 1 00000100:00100000:2.0:1789760756.635677:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745932738852 time 745966607999 state 0x0 ref 1 00000100:00100000:2.0:1789760756.635689:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706e050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484273920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760756.635739:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22630400, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:2.0:1789760756.635748:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.635763:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=745s v=(0 0) 00010000:00010000:2.0:1789760756.635768:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760756.635774:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22630400->22634495] (req 22630400->22634495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.635788:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22630400->22634495] (req 22630400->22634495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760756.674276:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22630400 -> 22634496 00010000:00010000:2.0:1789760756.674295:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22630400->22634495] (req 22630400->22634495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.674307:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22630400->22634495] (req 22630400->22634495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.674315:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22630400->22634495] (req 22630400->22634495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.674322:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760756.674326:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22630400->22634495] (req 22630400->22634495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.674334:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22630400->22634495] (req 22630400->22634495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.674356:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22630400->22634495] (req 22630400->22634495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.674363:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22630400->22634495] (req 22630400->22634495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760756.674572:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706e050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484273920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38879us (39117us total) trans 4294972843 rc 0/0 00000100:00100000:2.0:1789760756.674586:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5554 00000100:00100000:2.0:1789760756.674596:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745932738852 time 745966607999 state 0x2 ref 0 00000001:00080000:0.0:1789760756.675639:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972843 is committed 00000100:00100000:0.0:1789760756.680424:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760756.681347:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484274048 00000100:00100000:2.0:1789760756.681359:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484274048 00000100:00100000:2.0:1789760756.681485:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745932738852 time 745966607999 state 0x0 ref 1 00000100:00100000:2.0:1789760756.681501:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745966868415 time 745966607999 state 0x1 ref 1 00000100:00100000:2.0:1789760756.681537:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745966868415 time 746012467915 state 0x0 ref 1 00000100:00100000:2.0:1789760756.681552:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484274048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760756.681601:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22634496, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:2.0:1789760756.681610:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.681625:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:2.0:1789760756.681629:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760756.681635:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22634496->22638591] (req 22634496->22638591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.681649:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22634496->22638591] (req 22634496->22638591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760756.717988:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22634496 -> 22638592 00010000:00010000:2.0:1789760756.718018:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22634496->22638591] (req 22634496->22638591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.718032:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22634496->22638591] (req 22634496->22638591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.718041:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22634496->22638591] (req 22634496->22638591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.718048:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760756.718052:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22634496->22638591] (req 22634496->22638591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.718060:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22634496->22638591] (req 22634496->22638591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.718115:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22634496->22638591] (req 22634496->22638591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.718124:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090d7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22634496->22638591] (req 22634496->22638591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760756.718217:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484274048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36655us (37788us total) trans 4294972844 rc 0/0 00000100:00100000:2.0:1789760756.718227:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5555 00000100:00100000:2.0:1789760756.718237:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745966868415 time 746012467915 state 0x2 ref 0 00000001:00080000:3.0:1789760756.718358:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972844 is committed 00000100:00100000:3.0:1789760756.722244:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760756.722370:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484274176 00000100:00100000:0.0:1789760756.722380:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484274176 00000100:00100000:0.0:1789760756.722444:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 745966868415 time 746012467915 state 0x0 ref 1 00000100:00100000:0.0:1789760756.722459:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746012728331 time 746012467915 state 0x1 ref 1 00000100:00100000:0.0:1789760756.722490:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746012728331 time 746053420694 state 0x0 ref 1 00000100:00100000:0.0:1789760756.722505:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484274176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760756.722553:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22638592, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:0.0:1789760756.722562:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.722579:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:0.0:1789760756.722584:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760756.722590:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22638592->22642687] (req 22638592->22642687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760756.722603:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22638592->22642687] (req 22638592->22642687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760756.772491:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972845 is committed 00002000:00010000:3.0:1789760756.772553:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22638592 -> 22642688 00010000:00010000:3.0:1789760756.772568:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22638592->22642687] (req 22638592->22642687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.772581:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22638592->22642687] (req 22638592->22642687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.772591:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22638592->22642687] (req 22638592->22642687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.772598:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760756.772603:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22638592->22642687] (req 22638592->22642687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.772612:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22638592->22642687] (req 22638592->22642687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.772630:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22638592->22642687] (req 22638592->22642687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.772638:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962090de lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22638592->22642687] (req 22638592->22642687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760756.772730:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484274176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50225us (50493us total) trans 4294972845 rc 0/0 00000100:00100000:3.0:1789760756.772741:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5556 00000100:00100000:3.0:1789760756.772751:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746012728331 time 746053420694 state 0x2 ref 0 00000100:00100000:0.0:1789760756.776381:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760756.776525:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484274304 00000100:00100000:2.0:1789760756.776535:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484274304 00000100:00100000:2.0:1789760756.776590:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746012728331 time 746053420694 state 0x0 ref 1 00000100:00100000:2.0:1789760756.776606:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746053681110 time 746053420694 state 0x1 ref 1 00000100:00100000:2.0:1789760756.776670:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746053681110 time 746107600338 state 0x0 ref 1 00000100:00100000:2.0:1789760756.776685:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484274304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760756.776734:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22642688, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:2.0:1789760756.776743:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.776759:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:2.0:1789760756.776763:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760756.776770:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22642688->22646783] (req 22642688->22646783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.776784:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22642688->22646783] (req 22642688->22646783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760756.836356:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22642688 -> 22646784 00010000:00010000:3.0:1789760756.836375:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22642688->22646783] (req 22642688->22646783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.836389:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22642688->22646783] (req 22642688->22646783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.836398:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22642688->22646783] (req 22642688->22646783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.836407:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760756.836411:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22642688->22646783] (req 22642688->22646783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.836421:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22642688->22646783] (req 22642688->22646783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.836445:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22642688->22646783] (req 22642688->22646783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.836454:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962090e5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22642688->22646783] (req 22642688->22646783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760756.836542:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484274304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59855us (60165us total) trans 4294972846 rc 0/0 00000100:00100000:3.0:1789760756.836552:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5557 00000100:00100000:3.0:1789760756.836562:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746053681110 time 746107600338 state 0x2 ref 0 00000001:00080000:1.0:1789760756.836979:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972846 is committed 00000100:00100000:1.0:1789760756.840744:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760756.840850:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484274432 00000100:00100000:2.0:1789760756.840859:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484274432 00000100:00100000:2.0:1789760756.840926:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746053681110 time 746107600338 state 0x0 ref 1 00000100:00100000:2.0:1789760756.840942:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746107860754 time 746107600338 state 0x1 ref 1 00000100:00100000:2.0:1789760756.840972:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746107860754 time 746171903254 state 0x0 ref 1 00000100:00100000:2.0:1789760756.840989:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484274432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760756.841039:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22646784, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:2.0:1789760756.841048:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.841109:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:2.0:1789760756.841114:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760756.841121:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22646784->22650879] (req 22646784->22650879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.841136:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22646784->22650879] (req 22646784->22650879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760756.888286:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972847 is committed 00002000:00010000:2.0:1789760756.888351:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22646784 -> 22650880 00010000:00010000:2.0:1789760756.888367:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22646784->22650879] (req 22646784->22650879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.888380:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22646784->22650879] (req 22646784->22650879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.888389:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22646784->22650879] (req 22646784->22650879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.888396:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760756.888401:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22646784->22650879] (req 22646784->22650879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.888410:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22646784->22650879] (req 22646784->22650879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.888430:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22646784->22650879] (req 22646784->22650879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.888438:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962090ec lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22646784->22650879] (req 22646784->22650879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760756.888521:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484274432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47529us (47780us total) trans 4294972847 rc 0/0 00000100:00100000:2.0:1789760756.888531:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5558 00000100:00100000:2.0:1789760756.888542:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746107860754 time 746171903254 state 0x2 ref 0 00000100:00100000:0.0:1789760756.892871:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760756.893104:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484274560 00000100:00100000:3.0:1789760756.893115:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484274560 00000100:00100000:3.0:1789760756.893170:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746107860754 time 746171903254 state 0x0 ref 1 00000100:00100000:3.0:1789760756.893186:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746172163670 time 746171903254 state 0x1 ref 1 00000100:00100000:3.0:1789760756.893223:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746172163670 time 746224153868 state 0x0 ref 1 00000100:00100000:3.0:1789760756.893236:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb5880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484274560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760756.893284:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22650880, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:3.0:1789760756.893293:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.894155:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:3.0:1789760756.894160:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760756.894167:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22650880->22654975] (req 22650880->22654975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.894181:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22650880->22654975] (req 22650880->22654975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760756.925846:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22650880 -> 22654976 00010000:00010000:2.0:1789760756.925866:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22650880->22654975] (req 22650880->22654975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.925880:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22650880->22654975] (req 22650880->22654975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.925888:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22650880->22654975] (req 22650880->22654975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.925896:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760756.925901:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22650880->22654975] (req 22650880->22654975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.925910:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22650880->22654975] (req 22650880->22654975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.925931:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22650880->22654975] (req 22650880->22654975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.925937:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962090f3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22650880->22654975] (req 22650880->22654975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760756.926040:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb5880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484274560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32801us (33175us total) trans 4294972848 rc 0/0 00000100:00100000:2.0:1789760756.926051:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5559 00000100:00100000:2.0:1789760756.926061:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746172163670 time 746224153868 state 0x2 ref 0 00000001:00080000:2.0:1789760756.926138:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972848 is committed 00000100:00100000:0.0:1789760756.932872:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760756.933006:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484274688 00000100:00100000:3.0:1789760756.933018:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484274688 00000100:00100000:3.0:1789760756.933113:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746172163670 time 746224153868 state 0x0 ref 1 00000100:00100000:3.0:1789760756.933129:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746224414284 time 746224153868 state 0x1 ref 1 00000100:00100000:3.0:1789760756.933159:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746224414284 time 746264089604 state 0x0 ref 1 00000100:00100000:3.0:1789760756.933171:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484274688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760756.933230:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22654976, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:3.0:1789760756.933239:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.933254:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:3.0:1789760756.933259:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760756.933265:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22654976->22659071] (req 22654976->22659071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760756.933278:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22654976->22659071] (req 22654976->22659071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760756.983741:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972849 is committed 00002000:00010000:2.0:1789760756.983841:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22654976 -> 22659072 00010000:00010000:2.0:1789760756.983856:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22654976->22659071] (req 22654976->22659071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.983868:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22654976->22659071] (req 22654976->22659071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.983877:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22654976->22659071] (req 22654976->22659071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.983890:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760756.983904:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22654976->22659071] (req 22654976->22659071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.983914:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22654976->22659071] (req 22654976->22659071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.983933:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22654976->22659071] (req 22654976->22659071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.983941:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962090fa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22654976->22659071] (req 22654976->22659071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760756.984221:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484274688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51046us (51355us total) trans 4294972849 rc 0/0 00000100:00100000:2.0:1789760756.984231:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5560 00000100:00100000:2.0:1789760756.984240:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746224414284 time 746264089604 state 0x2 ref 0 00000100:00100000:1.0:1789760756.988475:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760756.988543:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484274816 00000100:00100000:2.0:1789760756.988553:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484274816 00000100:00100000:2.0:1789760756.988604:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746224414284 time 746264089604 state 0x0 ref 1 00000100:00100000:2.0:1789760756.988618:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746264350020 time 746264089604 state 0x1 ref 1 00000100:00100000:2.0:1789760756.988655:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746264350020 time 746319585281 state 0x0 ref 1 00000100:00100000:2.0:1789760756.988670:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484274816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760756.988715:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22659072, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:2.0:1789760756.988723:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.988737:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:2.0:1789760756.988742:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760756.988748:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22659072->22663167] (req 22659072->22663167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760756.988761:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22659072->22663167] (req 22659072->22663167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760757.059390:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972850 is committed 00002000:00010000:2.0:1789760757.059501:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22659072 -> 22663168 00010000:00010000:2.0:1789760757.059517:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22659072->22663167] (req 22659072->22663167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.059530:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22659072->22663167] (req 22659072->22663167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.059539:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22659072->22663167] (req 22659072->22663167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.059547:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760757.059552:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22659072->22663167] (req 22659072->22663167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.059562:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22659072->22663167] (req 22659072->22663167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.059581:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22659072->22663167] (req 22659072->22663167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.059589:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209101 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22659072->22663167] (req 22659072->22663167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760757.059681:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484274816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 71008us (71212us total) trans 4294972850 rc 0/0 00000100:00100000:2.0:1789760757.059691:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5561 00000100:00100000:2.0:1789760757.059701:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746264350020 time 746319585281 state 0x2 ref 0 00000100:00100000:0.0:1789760757.063940:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760757.064298:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484274944 00000100:00100000:2.0:1789760757.064308:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484274944 00000100:00100000:2.0:1789760757.064362:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746264350020 time 746319585281 state 0x0 ref 1 00000100:00100000:2.0:1789760757.064378:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746319845697 time 746319585281 state 0x1 ref 1 00000100:00100000:2.0:1789760757.064413:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746319845697 time 746395343587 state 0x0 ref 1 00000100:00100000:2.0:1789760757.064426:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484274944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760757.064482:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22663168, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:2.0:1789760757.064491:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.064508:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:2.0:1789760757.064513:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760757.064520:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22663168->22667263] (req 22663168->22667263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.064536:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22663168->22667263] (req 22663168->22667263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760757.110320:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972851 is committed 00002000:00010000:1.0:1789760757.110450:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22663168 -> 22667264 00010000:00010000:1.0:1789760757.110466:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22663168->22667263] (req 22663168->22667263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.110479:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22663168->22667263] (req 22663168->22667263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.110488:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22663168->22667263] (req 22663168->22667263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.110496:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760757.110500:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22663168->22667263] (req 22663168->22667263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.110510:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22663168->22667263] (req 22663168->22667263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.110532:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22663168->22667263] (req 22663168->22667263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.110541:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209108 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22663168->22667263] (req 22663168->22667263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760757.110634:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484274944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46206us (46699us total) trans 4294972851 rc 0/0 00000100:00100000:1.0:1789760757.110645:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5562 00000100:00100000:1.0:1789760757.110656:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746319845697 time 746395343587 state 0x2 ref 0 00000100:00100000:0.0:1789760757.116521:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760757.116675:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484275072 00000100:00100000:2.0:1789760757.116685:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484275072 00000100:00100000:2.0:1789760757.116736:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746319845697 time 746395343587 state 0x0 ref 1 00000100:00100000:2.0:1789760757.116750:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746395604003 time 746395343587 state 0x1 ref 1 00000100:00100000:2.0:1789760757.116781:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746395604003 time 746447711848 state 0x0 ref 1 00000100:00100000:2.0:1789760757.116793:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484275072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760757.116845:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22667264, mtime: 0x6aad94f4, atime: 0x0, ctime: 0x6aad94f4, blocks: 0xb007 00010000:00010000:2.0:1789760757.116855:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.116870:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:2.0:1789760757.116874:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760757.116880:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22667264->22671359] (req 22667264->22671359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.116893:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22667264->22671359] (req 22667264->22671359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760757.133365:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508927104:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.133436:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.134624:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508927616:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.134918:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.134995:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508928128:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.135024:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.135042:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508928640:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.135061:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.135099:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [2->3] 00000100:00100000:1.0:1789760757.135106:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b82300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508927488:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.135143:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.135162:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b83100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508928000:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.135181:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.135208:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508928512:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.135230:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.135263:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508927232:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.135284:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.135301:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508927744:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.135321:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.135339:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508928256:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.135378:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.135396:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfbb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508928768:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760757.135416:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.135434:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:1.0:1789760757.135440:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508927360:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.135461:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.136866:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508927872:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.136905:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.136920:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508928384:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.136942:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.136961:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfbb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508928896:0@lo:13:osp-pre-1-1.0 00000100:00100000:1.0:1789760757.136994:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760757.137250:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508927104 00000100:00100000:1.0:1789760757.137258:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508927104 00000100:00100000:1.0:1789760757.137306:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 484 00000100:00100000:1.0:1789760757.137315:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c81b5050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508927104:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.137370:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c81b5050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3481:x1876699508927104:12345-0@lo:400:kworker.0 Request processed in 54us (3933us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.137376:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 484 00000100:00100000:1.0:1789760757.137439:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508928128 00000100:00100000:1.0:1789760757.137443:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508928128 00000100:00100000:1.0:1789760757.137476:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508928640 00000100:00100000:1.0:1789760757.137479:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508928640 00000100:00100000:1.0:1789760757.137493:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508928000 00000100:00100000:1.0:1789760757.137495:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508928000 00000100:00100000:1.0:1789760757.137506:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508928512 00000100:00100000:1.0:1789760757.137509:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508928512 00000100:00100000:1.0:1789760757.137525:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 329 00000100:00100000:1.0:1789760757.137532:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9883b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508928128:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.137569:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9883b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508928128:12345-0@lo:400:kworker.0 Request processed in 37us (2543us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.137575:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 329 00000100:00100000:1.0:1789760757.137588:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 330 00000100:00100000:1.0:1789760757.137594:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9882d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508928640:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.137621:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9882d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508928640:12345-0@lo:400:kworker.0 Request processed in 28us (2559us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.137628:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 330 00000100:00100000:1.0:1789760757.137639:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 331 00000100:00100000:1.0:1789760757.137644:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9881f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508928000:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.137670:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9881f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508928000:12345-0@lo:400:kworker.0 Request processed in 26us (2487us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.137676:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 331 00000100:00100000:1.0:1789760757.137685:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 332 00000100:00100000:1.0:1789760757.137691:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9883480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508928512:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.137716:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9883480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508928512:12345-0@lo:400:kworker.0 Request processed in 25us (2486us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.137722:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 332 00000100:00100000:1.0:1789760757.137746:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508927488 00000100:00100000:1.0:1789760757.137750:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508927488 00000100:00100000:1.0:1789760757.137767:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508927232 00000100:00100000:1.0:1789760757.137769:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508927232 00000100:00100000:1.0:1789760757.137779:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508927744 00000100:00100000:1.0:1789760757.137782:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508927744 00000100:00100000:1.0:1789760757.137793:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508928256 00000100:00100000:1.0:1789760757.137795:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508928256 00000100:00100000:1.0:1789760757.137805:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508927872 00000100:00100000:1.0:1789760757.137807:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508927872 00000100:00100000:1.0:1789760757.137819:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508928384 00000100:00100000:1.0:1789760757.137821:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508928384 00000100:00100000:1.0:1789760757.137833:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 945 00000100:00100000:1.0:1789760757.137839:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9881500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508927488:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.137869:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9881500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508927488:12345-0@lo:400:kworker.0 Request processed in 30us (2725us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.137875:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 945 00000100:00100000:1.0:1789760757.137886:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 946 00000100:00100000:1.0:1789760757.137891:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9883800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508927232:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.137920:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9883800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508927232:12345-0@lo:400:kworker.0 Request processed in 28us (2635us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.137926:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 946 00000100:00100000:1.0:1789760757.137935:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 947 00000100:00100000:1.0:1789760757.137940:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9883100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508927744:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.137963:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9883100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508927744:12345-0@lo:400:kworker.0 Request processed in 23us (2641us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.137968:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 947 00000100:00100000:1.0:1789760757.137978:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 948 00000100:00100000:1.0:1789760757.138139:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9880e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508928256:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.138170:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9880e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508928256:12345-0@lo:400:kworker.0 Request processed in 33us (2795us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760757.138263:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508927616 00000100:00100000:0.0:1789760757.138268:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508927616 00000100:00100000:0.0:1789760757.138302:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508927360 00000100:00100000:0.0:1789760757.138306:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508927360 00000100:00100000:0.0:1789760757.138319:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 451 00000100:00100000:0.0:1789760757.138326:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508927616:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760757.138361:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508927616:12345-0@lo:400:kworker.0 Request processed in 36us (3444us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760757.138367:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 451 00000100:00100000:0.0:1789760757.138379:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 452 00000100:00100000:0.0:1789760757.138386:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9882680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508927360:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760757.138409:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9882680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508927360:12345-0@lo:400:kworker.0 Request processed in 25us (2947us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760757.138415:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 452 00000100:00100000:3.0:1789760757.138720:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 949 00000100:00100000:3.0:1789760757.138732:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9881c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508927872:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760757.138767:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9881c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508927872:12345-0@lo:400:kworker.0 Request processed in 40us (1862us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760757.138774:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 949 00000100:00100000:3.0:1789760757.138789:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 950 00000100:00100000:3.0:1789760757.138794:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a32a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508928384:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760757.138819:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a32a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508928384:12345-0@lo:400:kworker.0 Request processed in 26us (1877us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760757.138825:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 950 00000100:00100000:1.0:1789760757.138874:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 948 00000100:00100000:1.0:1789760757.138918:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508928768 00000100:00100000:1.0:1789760757.138922:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508928768 00000100:00100000:1.0:1789760757.138956:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508928896 00000100:00100000:1.0:1789760757.138960:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508928896 00000100:00100000:1.0:1789760757.138972:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 392 00000100:00100000:1.0:1789760757.138978:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9882300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508928768:12345-0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760757.139092:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 393 00000100:00100000:0.0:1789760757.139102:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508928896:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:0.0:1789760757.139133:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:0.0:1789760757.139167:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508928896:12345-0@lo:13:osp-pre-1-1.0 Request processed in 69us (2173us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760757.139173:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 393 00002000:00000024:1.0:1789760757.139256:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760757.139290:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9882300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508928768:12345-0@lo:13:osp-pre-1-0.0 Request processed in 311us (3872us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.139297:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 392 00000100:00080000:1.0:1789760757.139327:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:1.0:1789760757.139331:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:1.0:1789760757.139334:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:1.0:1789760757.139336:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1789760757.139375:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b82680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508927360:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139388:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b81500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508927872:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139398:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b81c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508928384:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139414:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfbb4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508928896:0@lo:13:osp-pre-1-1.0 00000100:00100000:1.0:1789760757.139425:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b83800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508927232:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139435:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b83480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508927744:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139445:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b82a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508928256:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139459:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfbb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508928768:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760757.139510:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b82300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508927488:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139521:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b83100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508928000:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139530:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508928512:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139538:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3481:1876699508927104:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139548:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508927616:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139559:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508928128:0@lo:400:kworker.0 00000100:00100000:1.0:1789760757.139567:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508928640:0@lo:400:kworker.0 00002000:00010000:3.0:1789760757.168428:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22667264 -> 22671360 00002000:00010000:3.0:1789760757.168438:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760756 -> 1789760757 00002000:00010000:3.0:1789760757.168441:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760756 -> 1789760757 00010000:00010000:3.0:1789760757.168455:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22667264->22671359] (req 22667264->22671359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.168469:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22667264->22671359] (req 22667264->22671359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.168478:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22667264->22671359] (req 22667264->22671359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.168491:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760757.168496:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22667264->22671359] (req 22667264->22671359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.168505:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22667264->22671359] (req 22667264->22671359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.168525:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22667264->22671359] (req 22667264->22671359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.168534:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620910f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22667264->22671359] (req 22667264->22671359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760757.168637:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484275072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51841us (52121us total) trans 4294972852 rc 0/0 00000100:00100000:3.0:1789760757.168648:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5563 00000100:00100000:3.0:1789760757.168658:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746395604003 time 746447711848 state 0x2 ref 0 00000001:00080000:2.0:1789760757.170328:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972852 is committed 00000100:00100000:1.0:1789760757.173432:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760757.173851:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484275200 00000100:00100000:3.0:1789760757.173862:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484275200 00000100:00100000:3.0:1789760757.173923:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746395604003 time 746447711848 state 0x0 ref 1 00000100:00100000:3.0:1789760757.173940:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746447972264 time 746447711848 state 0x1 ref 1 00000100:00100000:3.0:1789760757.173974:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746447972264 time 746504904662 state 0x0 ref 1 00000100:00100000:3.0:1789760757.173998:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484275200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760757.174049:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22671360, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:3.0:1789760757.174058:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.174102:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:3.0:1789760757.174107:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760757.174114:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22671360->22675455] (req 22671360->22675455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.174129:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22671360->22675455] (req 22671360->22675455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760757.231640:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972853 is committed 00002000:00010000:3.0:1789760757.231983:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22671360 -> 22675456 00010000:00010000:3.0:1789760757.232001:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22671360->22675455] (req 22671360->22675455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.232013:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22671360->22675455] (req 22671360->22675455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.232022:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22671360->22675455] (req 22671360->22675455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.232029:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760757.232033:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22671360->22675455] (req 22671360->22675455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.232042:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22671360->22675455] (req 22671360->22675455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.232084:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22671360->22675455] (req 22671360->22675455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.232093:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209116 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22671360->22675455] (req 22671360->22675455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760757.232186:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484275200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58186us (58759us total) trans 4294972853 rc 0/0 00000100:00100000:3.0:1789760757.232206:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5564 00000100:00100000:3.0:1789760757.232216:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746447972264 time 746504904662 state 0x2 ref 0 00000100:00100000:0.0:1789760757.236280:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760757.236358:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484275328 00000100:00100000:3.0:1789760757.236368:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484275328 00000100:00100000:3.0:1789760757.236421:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746447972264 time 746504904662 state 0x0 ref 1 00000100:00100000:3.0:1789760757.236435:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746505165078 time 746504904662 state 0x1 ref 1 00000100:00100000:3.0:1789760757.236467:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746505165078 time 746567398446 state 0x0 ref 1 00000100:00100000:3.0:1789760757.236478:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484275328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760757.236526:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22675456, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:3.0:1789760757.236534:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.236549:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:3.0:1789760757.236554:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760757.236560:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22675456->22679551] (req 22675456->22679551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.236572:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22675456->22679551] (req 22675456->22679551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760757.309486:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972854 is committed 00002000:00010000:3.0:1789760757.310168:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22675456 -> 22679552 00010000:00010000:3.0:1789760757.310185:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22675456->22679551] (req 22675456->22679551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.310206:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22675456->22679551] (req 22675456->22679551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.310215:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22675456->22679551] (req 22675456->22679551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.310223:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760757.310228:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22675456->22679551] (req 22675456->22679551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.310237:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22675456->22679551] (req 22675456->22679551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.310256:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22675456->22679551] (req 22675456->22679551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.310264:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620911d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22675456->22679551] (req 22675456->22679551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760757.310372:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484275328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73890us (74096us total) trans 4294972854 rc 0/0 00000100:00100000:3.0:1789760757.310383:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5565 00000100:00100000:3.0:1789760757.310393:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746505165078 time 746567398446 state 0x2 ref 0 00000100:00100000:0.0:1789760757.315876:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760757.316255:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484275456 00000100:00100000:1.0:1789760757.316266:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484275456 00000100:00100000:1.0:1789760757.316328:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746505165078 time 746567398446 state 0x0 ref 1 00000100:00100000:1.0:1789760757.316344:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746567658862 time 746567398446 state 0x1 ref 1 00000100:00100000:1.0:1789760757.316374:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746567658862 time 746647305042 state 0x0 ref 1 00000100:00100000:1.0:1789760757.316388:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484275456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760757.316440:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22679552, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:1.0:1789760757.316449:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.316465:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:1.0:1789760757.316470:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760757.316476:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22679552->22683647] (req 22679552->22683647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.316491:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22679552->22683647] (req 22679552->22683647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760757.361508:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972855 is committed 00002000:00010000:3.0:1789760757.361559:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22679552 -> 22683648 00010000:00010000:3.0:1789760757.361575:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22679552->22683647] (req 22679552->22683647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.361589:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22679552->22683647] (req 22679552->22683647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.361598:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22679552->22683647] (req 22679552->22683647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.361605:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760757.361610:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22679552->22683647] (req 22679552->22683647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.361620:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22679552->22683647] (req 22679552->22683647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.361645:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22679552->22683647] (req 22679552->22683647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.361654:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209124 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22679552->22683647] (req 22679552->22683647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760757.361755:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484275456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45365us (45885us total) trans 4294972855 rc 0/0 00000100:00100000:3.0:1789760757.361765:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5566 00000100:00100000:3.0:1789760757.361783:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746567658862 time 746647305042 state 0x2 ref 0 00000100:00100000:0.0:1789760757.371291:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760757.371522:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484275584 00000100:00100000:1.0:1789760757.371531:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484275584 00000100:00100000:1.0:1789760757.371585:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746567658862 time 746647305042 state 0x0 ref 1 00000100:00100000:1.0:1789760757.371600:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746647565458 time 746647305042 state 0x1 ref 1 00000100:00100000:1.0:1789760757.371631:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746647565458 time 746702562145 state 0x0 ref 1 00000100:00100000:1.0:1789760757.371643:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484275584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760757.371697:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22683648, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:1.0:1789760757.371705:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.371721:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:1.0:1789760757.371725:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760757.371731:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22683648->22687743] (req 22683648->22687743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.371745:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22683648->22687743] (req 22683648->22687743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760757.419650:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972856 is committed 00002000:00010000:0.0:1789760757.419707:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22683648 -> 22687744 00010000:00010000:0.0:1789760757.419727:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22683648->22687743] (req 22683648->22687743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.419737:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22683648->22687743] (req 22683648->22687743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.419743:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22683648->22687743] (req 22683648->22687743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.419748:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760757.419751:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22683648->22687743] (req 22683648->22687743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.419758:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22683648->22687743] (req 22683648->22687743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.419777:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22683648->22687743] (req 22683648->22687743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.419781:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620912b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22683648->22687743] (req 22683648->22687743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760757.419867:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484275584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48214us (48573us total) trans 4294972856 rc 0/0 00000100:00100000:0.0:1789760757.419877:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5567 00000100:00100000:0.0:1789760757.419887:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746647565458 time 746702562145 state 0x2 ref 0 00000100:00100000:0.0:1789760757.426212:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760757.426733:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484275712 00000100:00100000:3.0:1789760757.426743:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484275712 00000100:00100000:3.0:1789760757.426868:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746647565458 time 746702562145 state 0x0 ref 1 00000100:00100000:3.0:1789760757.426885:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746702822561 time 746702562145 state 0x1 ref 1 00000100:00100000:3.0:1789760757.426919:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746702822561 time 746757850073 state 0x0 ref 1 00000100:00100000:3.0:1789760757.426933:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484275712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760757.427057:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22687744, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:3.0:1789760757.427095:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.427113:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:3.0:1789760757.427119:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760757.427125:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22687744->22691839] (req 22687744->22691839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.427139:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22687744->22691839] (req 22687744->22691839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760757.470820:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972857 is committed 00002000:00010000:3.0:1789760757.471312:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22687744 -> 22691840 00010000:00010000:3.0:1789760757.471329:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22687744->22691839] (req 22687744->22691839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.471342:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22687744->22691839] (req 22687744->22691839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.471351:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22687744->22691839] (req 22687744->22691839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.471359:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760757.471363:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22687744->22691839] (req 22687744->22691839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.471372:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22687744->22691839] (req 22687744->22691839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.471392:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22687744->22691839] (req 22687744->22691839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.471399:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209132 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22687744->22691839] (req 22687744->22691839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760757.471525:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484275712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44590us (45327us total) trans 4294972857 rc 0/0 00000100:00100000:3.0:1789760757.471535:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5568 00000100:00100000:3.0:1789760757.471545:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746702822561 time 746757850073 state 0x2 ref 0 00000100:00100000:0.0:1789760757.479383:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760757.479481:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484275840 00000100:00100000:3.0:1789760757.479492:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484275840 00000100:00100000:3.0:1789760757.479544:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746702822561 time 746757850073 state 0x0 ref 1 00000100:00100000:3.0:1789760757.479559:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746758110489 time 746757850073 state 0x1 ref 1 00000100:00100000:3.0:1789760757.479588:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746758110489 time 746810519242 state 0x0 ref 1 00000100:00100000:3.0:1789760757.479600:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484275840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760757.479650:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22691840, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:3.0:1789760757.479812:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.479829:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:3.0:1789760757.479834:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760757.479840:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22691840->22695935] (req 22691840->22695935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.479865:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22691840->22695935] (req 22691840->22695935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760757.531255:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972858 is committed 00002000:00010000:3.0:1789760757.531330:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22691840 -> 22695936 00010000:00010000:3.0:1789760757.531344:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22691840->22695935] (req 22691840->22695935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.531356:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22691840->22695935] (req 22691840->22695935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.531364:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22691840->22695935] (req 22691840->22695935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.531370:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760757.531374:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22691840->22695935] (req 22691840->22695935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.531382:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22691840->22695935] (req 22691840->22695935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.531401:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22691840->22695935] (req 22691840->22695935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.531407:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209139 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22691840->22695935] (req 22691840->22695935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760757.531526:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484275840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51924us (52149us total) trans 4294972858 rc 0/0 00000100:00100000:3.0:1789760757.531538:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5569 00000100:00100000:3.0:1789760757.531546:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746758110489 time 746810519242 state 0x2 ref 0 00000100:00100000:0.0:1789760757.542622:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760757.542703:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484275968 00000100:00100000:1.0:1789760757.542712:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484275968 00000100:00100000:1.0:1789760757.542764:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746758110489 time 746810519242 state 0x0 ref 1 00000100:00100000:1.0:1789760757.542780:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746810779658 time 746810519242 state 0x1 ref 1 00000100:00100000:1.0:1789760757.542812:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746810779658 time 746873742803 state 0x0 ref 1 00000100:00100000:1.0:1789760757.542825:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484275968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760757.542883:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22695936, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:1.0:1789760757.542892:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.542907:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:1.0:1789760757.542911:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760757.542916:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22695936->22700031] (req 22695936->22700031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.542929:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22695936->22700031] (req 22695936->22700031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760757.605889:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972859 is committed 00002000:00010000:3.0:1789760757.606057:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22695936 -> 22700032 00010000:00010000:3.0:1789760757.606101:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22695936->22700031] (req 22695936->22700031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.606115:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22695936->22700031] (req 22695936->22700031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.606125:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22695936->22700031] (req 22695936->22700031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.606133:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760757.606138:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22695936->22700031] (req 22695936->22700031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.606148:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22695936->22700031] (req 22695936->22700031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.606167:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22695936->22700031] (req 22695936->22700031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.606175:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209140 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22695936->22700031] (req 22695936->22700031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760757.606278:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484275968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63450us (63661us total) trans 4294972859 rc 0/0 00000100:00100000:3.0:1789760757.606290:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5570 00000100:00100000:3.0:1789760757.606300:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746810779658 time 746873742803 state 0x2 ref 0 00000100:00100000:0.0:1789760757.615862:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760757.619025:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484276096 00000100:00100000:1.0:1789760757.619040:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484276096 00000100:00100000:0.0:1789760757.621949:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746810779658 time 746873742803 state 0x0 ref 1 00000100:00100000:0.0:1789760757.621967:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746874003219 time 746873742803 state 0x1 ref 1 00000100:00100000:0.0:1789760757.622011:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746874003219 time 746952941050 state 0x0 ref 1 00000100:00100000:0.0:1789760757.622028:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484276096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760757.622187:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22700032, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:0.0:1789760757.622205:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.622221:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:0.0:1789760757.622226:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760757.622232:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22700032->22704127] (req 22700032->22704127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.622248:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22700032->22704127] (req 22700032->22704127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.648859:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760757.651313:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972860 is committed 00002000:00010000:3.0:1789760757.651446:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22700032 -> 22704128 00010000:00010000:3.0:1789760757.651461:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22700032->22704127] (req 22700032->22704127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.651474:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22700032->22704127] (req 22700032->22704127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.651483:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22700032->22704127] (req 22700032->22704127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.651491:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760757.651495:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22700032->22704127] (req 22700032->22704127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.651504:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22700032->22704127] (req 22700032->22704127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.651522:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22700032->22704127] (req 22700032->22704127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.651551:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209147 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22700032->22704127] (req 22700032->22704127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760757.651676:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484276096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29648us (35821us total) trans 4294972860 rc 0/0 00000100:00100000:3.0:1789760757.651686:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5571 00000100:00100000:3.0:1789760757.651696:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746874003219 time 746952941050 state 0x2 ref 0 00000100:00100000:0.0:1789760757.664594:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760757.664681:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484276224 00000100:00100000:3.0:1789760757.664691:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484276224 00000100:00100000:3.0:1789760757.664741:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746874003219 time 746952941050 state 0x0 ref 1 00000100:00100000:3.0:1789760757.664756:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746953201466 time 746952941050 state 0x1 ref 1 00000100:00100000:3.0:1789760757.664790:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746953201466 time 746995721034 state 0x0 ref 1 00000100:00100000:3.0:1789760757.664803:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484276224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760757.664849:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22704128, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:3.0:1789760757.664857:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.664873:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=746s v=(0 0) 00010000:00010000:3.0:1789760757.664877:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760757.664884:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22704128->22708223] (req 22704128->22708223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.664904:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22704128->22708223] (req 22704128->22708223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760757.714633:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972861 is committed 00002000:00010000:3.0:1789760757.715396:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22704128 -> 22708224 00010000:00010000:3.0:1789760757.715413:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22704128->22708223] (req 22704128->22708223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.715424:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22704128->22708223] (req 22704128->22708223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.715433:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22704128->22708223] (req 22704128->22708223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.715441:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760757.715445:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22704128->22708223] (req 22704128->22708223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.715454:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22704128->22708223] (req 22704128->22708223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.715475:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22704128->22708223] (req 22704128->22708223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.715482:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620914e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22704128->22708223] (req 22704128->22708223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760757.715587:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484276224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50781us (50997us total) trans 4294972861 rc 0/0 00000100:00100000:3.0:1789760757.715599:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5572 00000100:00100000:3.0:1789760757.715608:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746953201466 time 746995721034 state 0x2 ref 0 00000100:00100000:0.0:1789760757.722647:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760757.722713:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484276352 00000100:00100000:3.0:1789760757.722723:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484276352 00000100:00100000:3.0:1789760757.722777:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746953201466 time 746995721034 state 0x0 ref 1 00000100:00100000:3.0:1789760757.722791:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746995981450 time 746995721034 state 0x1 ref 1 00000100:00100000:3.0:1789760757.722819:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746995981450 time 747053750756 state 0x0 ref 1 00000100:00100000:3.0:1789760757.722831:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484276352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760757.722876:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22708224, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:3.0:1789760757.722885:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.722900:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:3.0:1789760757.722904:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760757.722909:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22708224->22712319] (req 22708224->22712319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.722922:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22708224->22712319] (req 22708224->22712319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760757.780361:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22708224 -> 22712320 00010000:00010000:0.0:1789760757.780381:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22708224->22712319] (req 22708224->22712319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.780395:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22708224->22712319] (req 22708224->22712319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.780404:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22708224->22712319] (req 22708224->22712319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.780412:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760757.780417:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22708224->22712319] (req 22708224->22712319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.780426:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22708224->22712319] (req 22708224->22712319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.780447:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22708224->22712319] (req 22708224->22712319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.780454:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209155 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22708224->22712319] (req 22708224->22712319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760757.780569:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484276352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57734us (57924us total) trans 4294972862 rc 0/0 00000100:00100000:0.0:1789760757.780580:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5573 00000100:00100000:0.0:1789760757.780590:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746995981450 time 747053750756 state 0x2 ref 0 00000001:00080000:0.0:1789760757.780632:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972862 is committed 00000100:00100000:0.0:1789760757.784727:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760757.784823:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484276480 00000100:00100000:1.0:1789760757.784833:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484276480 00000100:00100000:1.0:1789760757.784888:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 746995981450 time 747053750756 state 0x0 ref 1 00000100:00100000:1.0:1789760757.784906:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747054011172 time 747053750756 state 0x1 ref 1 00000100:00100000:1.0:1789760757.785037:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747054011172 time 747115966935 state 0x0 ref 1 00000100:00100000:1.0:1789760757.785050:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484276480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760757.785129:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22712320, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:1.0:1789760757.785138:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.785152:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:1.0:1789760757.785157:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760757.785163:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22712320->22716415] (req 22712320->22716415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760757.785178:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22712320->22716415] (req 22712320->22716415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760757.830109:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972863 is committed 00002000:00010000:0.0:1789760757.830118:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22712320 -> 22716416 00010000:00010000:0.0:1789760757.830136:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22712320->22716415] (req 22712320->22716415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.830148:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22712320->22716415] (req 22712320->22716415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.830158:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22712320->22716415] (req 22712320->22716415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.830165:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760757.830170:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22712320->22716415] (req 22712320->22716415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.830179:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22712320->22716415] (req 22712320->22716415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.830208:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22712320->22716415] (req 22712320->22716415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.830217:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620915c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22712320->22716415] (req 22712320->22716415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760757.830355:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484276480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45301us (45633us total) trans 4294972863 rc 0/0 00000100:00100000:0.0:1789760757.830366:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5574 00000100:00100000:0.0:1789760757.830376:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747054011172 time 747115966935 state 0x2 ref 0 00000100:00100000:2.0:1789760757.843411:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760757.843572:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484276608 00000100:00100000:0.0:1789760757.843583:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484276608 00000100:00100000:0.0:1789760757.843942:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747054011172 time 747115966935 state 0x0 ref 1 00000100:00100000:0.0:1789760757.843958:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747116227351 time 747115966935 state 0x1 ref 1 00000100:00100000:0.0:1789760757.843991:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747116227351 time 747174921622 state 0x0 ref 1 00000100:00100000:0.0:1789760757.844006:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ad880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484276608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760757.844054:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22716416, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:0.0:1789760757.844094:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.844109:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:0.0:1789760757.844113:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760757.844119:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22716416->22720511] (req 22716416->22720511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.844133:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22716416->22720511] (req 22716416->22720511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760757.927900:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22716416 -> 22720512 00010000:00010000:0.0:1789760757.927927:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22716416->22720511] (req 22716416->22720511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.927941:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22716416->22720511] (req 22716416->22720511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.928059:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22716416->22720511] (req 22716416->22720511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.928087:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760757.928090:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22716416->22720511] (req 22716416->22720511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.928097:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22716416->22720511] (req 22716416->22720511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.928114:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22716416->22720511] (req 22716416->22720511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.928119:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209163 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22716416->22720511] (req 22716416->22720511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760757.928146:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972864 is committed 00000100:00100000:0.0:1789760757.928213:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ad880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484276608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 84208us (84809us total) trans 4294972864 rc 0/0 00000100:00100000:0.0:1789760757.928224:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5575 00000100:00100000:0.0:1789760757.928233:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747116227351 time 747174921622 state 0x2 ref 0 00000100:00100000:2.0:1789760757.933335:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760757.933836:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484276736 00000100:00100000:3.0:1789760757.933846:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484276736 00000100:00100000:3.0:1789760757.933902:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747116227351 time 747174921622 state 0x0 ref 1 00000100:00100000:3.0:1789760757.933919:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747175182038 time 747174921622 state 0x1 ref 1 00000100:00100000:3.0:1789760757.933946:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747175182038 time 747264877450 state 0x0 ref 1 00000100:00100000:3.0:1789760757.933960:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484276736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760757.934129:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22720512, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:3.0:1789760757.934138:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.934155:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:3.0:1789760757.934159:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760757.934166:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22720512->22724607] (req 22720512->22724607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760757.934180:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22720512->22724607] (req 22720512->22724607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760757.974214:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972865 is committed 00002000:00010000:0.0:1789760757.974316:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22720512 -> 22724608 00010000:00010000:0.0:1789760757.974332:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22720512->22724607] (req 22720512->22724607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.974345:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22720512->22724607] (req 22720512->22724607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.974353:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22720512->22724607] (req 22720512->22724607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.974361:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760757.974367:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22720512->22724607] (req 22720512->22724607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.974376:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22720512->22724607] (req 22720512->22724607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.974395:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22720512->22724607] (req 22720512->22724607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760757.974403:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620916a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22720512->22724607] (req 22720512->22724607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760757.974498:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484276736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40536us (41167us total) trans 4294972865 rc 0/0 00000100:00100000:0.0:1789760757.974509:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5576 00000100:00100000:0.0:1789760757.974519:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747175182038 time 747264877450 state 0x2 ref 0 00000100:00100000:3.0:1789760757.980612:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760757.980769:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484276864 00000100:00100000:2.0:1789760757.980781:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484276864 00000100:00100000:2.0:1789760757.980835:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747175182038 time 747264877450 state 0x0 ref 1 00000100:00100000:2.0:1789760757.980852:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747265137866 time 747264877450 state 0x1 ref 1 00000100:00100000:2.0:1789760757.980936:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747265137866 time 747311865244 state 0x0 ref 1 00000100:00100000:2.0:1789760757.980950:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484276864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760757.981009:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22724608, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:2.0:1789760757.981019:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.981035:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:2.0:1789760757.981040:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760757.981046:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22724608->22728703] (req 22724608->22728703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760757.981061:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22724608->22728703] (req 22724608->22728703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760757.998803:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760757.998840:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760757.998865:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760757.998884:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760757.998904:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760757.998968:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484277120 00000100:00100000:3.0:1789760757.998978:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484277120 00000100:00100000:3.0:1789760757.999374:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484277376 00000100:00100000:3.0:1789760757.999380:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484277376 00000100:00100000:3.0:1789760757.999404:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484277248 00000100:00100000:3.0:1789760757.999407:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484277248 00000100:00100000:3.0:1789760757.999422:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484277504 00000100:00100000:3.0:1789760757.999424:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484277504 00000100:00080000:2.0:1789760757.999508:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760757.999513:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:3.0:1789760757.999588:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 951 00000100:00100000:3.0:1789760757.999601:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12aea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2422:x1876699484277120:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760757.999755:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12aea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2422:x1876699484277120:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 157us (962us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.999828:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484276992 00000100:00100000:1.0:1789760757.999832:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484276992 00000100:00100000:1.0:1789760757.999864:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 485 00000100:00100000:1.0:1789760757.999871:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8038050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484276992:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760757.999900:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8038050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484276992:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 30us (1033us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760757.999906:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 485 00000100:00100000:0.0:1789760758.000853:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 952 00000100:00100000:0.0:1789760758.000870:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ad500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484277376:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760758.000921:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ad500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484277376:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 58us (2080us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760758.000929:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 952 00000100:00100000:0.0:1789760758.000950:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 953 00000100:00100000:0.0:1789760758.000957:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ae300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484277248:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760758.001106:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 951 00000100:00100000:3.0:1789760758.001118:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 954 00000100:00100000:3.0:1789760758.001125:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ace00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484277504:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760758.001146:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ace00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484277504:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 23us (2242us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760758.001150:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 954 00000100:00100000:0.0:1789760758.001214:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ae300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484277248:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 257us (2328us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760758.001225:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 953 00000001:00080000:1.0:1789760758.029670:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972866 is committed 00002000:00010000:0.0:1789760758.029782:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22724608 -> 22728704 00010000:00010000:0.0:1789760758.029797:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22724608->22728703] (req 22724608->22728703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.029810:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22724608->22728703] (req 22724608->22728703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.029820:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22724608->22728703] (req 22724608->22728703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.029828:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760758.029833:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22724608->22728703] (req 22724608->22728703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.029842:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22724608->22728703] (req 22724608->22728703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.029861:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22724608->22728703] (req 22724608->22728703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.029867:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209171 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22724608->22728703] (req 22724608->22728703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760758.029996:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484276864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49043us (49389us total) trans 4294972866 rc 0/0 00000100:00100000:0.0:1789760758.030006:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5577 00000100:00100000:0.0:1789760758.030016:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747265137866 time 747311865244 state 0x2 ref 0 00000100:00100000:2.0:1789760758.038375:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760758.038478:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484277632 00000100:00100000:0.0:1789760758.038487:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484277632 00000100:00100000:0.0:1789760758.038558:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747265137866 time 747311865244 state 0x0 ref 1 00000100:00100000:0.0:1789760758.038574:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747312125660 time 747311865244 state 0x1 ref 1 00000100:00100000:0.0:1789760758.038605:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747312125660 time 747369535691 state 0x0 ref 1 00000100:00100000:0.0:1789760758.038624:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484277632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760758.038674:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22728704, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:0.0:1789760758.038683:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.038699:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:0.0:1789760758.038704:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760758.038710:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22728704->22732799] (req 22728704->22732799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.038723:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22728704->22732799] (req 22728704->22732799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760758.039102:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:1.0:1789760758.077220:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22728704 -> 22732800 00010000:00010000:1.0:1789760758.077242:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22728704->22732799] (req 22728704->22732799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.077256:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22728704->22732799] (req 22728704->22732799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.077265:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22728704->22732799] (req 22728704->22732799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.077273:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760758.077279:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22728704->22732799] (req 22728704->22732799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760758.077283:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972867 is committed 00010000:00010000:1.0:1789760758.077292:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22728704->22732799] (req 22728704->22732799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.077317:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22728704->22732799] (req 22728704->22732799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.077326:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209178 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22728704->22732799] (req 22728704->22732799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760758.077425:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484277632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38805us (39056us total) trans 4294972867 rc 0/0 00000100:00100000:1.0:1789760758.077437:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5578 00000100:00100000:1.0:1789760758.077448:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747312125660 time 747369535691 state 0x2 ref 0 00000100:00100000:2.0:1789760758.081668:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760758.081948:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484277760 00000100:00100000:1.0:1789760758.081958:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484277760 00000100:00100000:1.0:1789760758.082022:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747312125660 time 747369535691 state 0x0 ref 1 00000100:00100000:1.0:1789760758.082036:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747369796107 time 747369535691 state 0x1 ref 1 00000100:00100000:1.0:1789760758.082099:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747369796107 time 747413029878 state 0x0 ref 1 00000100:00100000:1.0:1789760758.082113:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12af800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484277760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760758.082158:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22732800, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:1.0:1789760758.082166:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.082180:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:1.0:1789760758.082185:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760758.082204:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22732800->22736895] (req 22732800->22736895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.082219:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22732800->22736895] (req 22732800->22736895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760758.136381:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972868 is committed 00002000:00010000:1.0:1789760758.137459:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22732800 -> 22736896 00010000:00010000:1.0:1789760758.137475:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22732800->22736895] (req 22732800->22736895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.137548:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22732800->22736895] (req 22732800->22736895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.137560:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22732800->22736895] (req 22732800->22736895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.137567:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760758.137572:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22732800->22736895] (req 22732800->22736895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.137581:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22732800->22736895] (req 22732800->22736895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.137600:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22732800->22736895] (req 22732800->22736895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.137608:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620917f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22732800->22736895] (req 22732800->22736895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760758.145223:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12af800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484277760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63099us (63552us total) trans 4294972868 rc 0/0 00000100:00100000:1.0:1789760758.145239:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5579 00000100:00100000:1.0:1789760758.145250:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747369796107 time 747413029878 state 0x2 ref 0 00000100:00100000:0.0:1789760758.153314:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760758.153476:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484277888 00000100:00100000:3.0:1789760758.153486:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484277888 00000100:00100000:3.0:1789760758.153546:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747369796107 time 747413029878 state 0x0 ref 1 00000100:00100000:3.0:1789760758.153562:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747413290294 time 747413029878 state 0x1 ref 1 00000100:00100000:3.0:1789760758.153594:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747413290294 time 747484525520 state 0x0 ref 1 00000100:00100000:3.0:1789760758.153606:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484277888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760758.153652:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22736896, mtime: 0x6aad94f5, atime: 0x0, ctime: 0x6aad94f5, blocks: 0xb007 00010000:00010000:3.0:1789760758.153662:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.153677:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:3.0:1789760758.153682:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760758.153688:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22736896->22740991] (req 22736896->22740991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.153701:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22736896->22740991] (req 22736896->22740991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760758.200787:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22736896 -> 22740992 00002000:00010000:3.0:1789760758.200795:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760757 -> 1789760758 00002000:00010000:3.0:1789760758.200798:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760757 -> 1789760758 00010000:00010000:3.0:1789760758.200811:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22736896->22740991] (req 22736896->22740991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.200825:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22736896->22740991] (req 22736896->22740991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.200835:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22736896->22740991] (req 22736896->22740991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.200853:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760758.200858:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22736896->22740991] (req 22736896->22740991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.200874:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22736896->22740991] (req 22736896->22740991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.200893:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22736896->22740991] (req 22736896->22740991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.200901:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209186 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22736896->22740991] (req 22736896->22740991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760758.201039:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484277888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47430us (47730us total) trans 4294972869 rc 0/0 00000100:00100000:3.0:1789760758.201051:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5580 00000100:00100000:3.0:1789760758.201062:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747413290294 time 747484525520 state 0x2 ref 0 00000001:00080000:2.0:1789760758.201166:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972869 is committed 00000100:00100000:2.0:1789760758.207452:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760758.207544:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484278016 00000100:00100000:0.0:1789760758.207553:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484278016 00000100:00100000:0.0:1789760758.207608:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747413290294 time 747484525520 state 0x0 ref 1 00000100:00100000:0.0:1789760758.207623:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747484785936 time 747484525520 state 0x1 ref 1 00000100:00100000:0.0:1789760758.207653:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747484785936 time 747538583854 state 0x0 ref 1 00000100:00100000:0.0:1789760758.207666:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ad180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484278016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760758.207711:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22740992, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:0.0:1789760758.207720:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.207737:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:0.0:1789760758.207742:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760758.207749:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22740992->22745087] (req 22740992->22745087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.207764:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22740992->22745087] (req 22740992->22745087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760758.253724:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972870 is committed 00002000:00010000:2.0:1789760758.253846:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22740992 -> 22745088 00010000:00010000:2.0:1789760758.253861:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22740992->22745087] (req 22740992->22745087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.253874:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22740992->22745087] (req 22740992->22745087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.253883:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22740992->22745087] (req 22740992->22745087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.253891:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760758.253896:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22740992->22745087] (req 22740992->22745087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.253906:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22740992->22745087] (req 22740992->22745087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.253924:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22740992->22745087] (req 22740992->22745087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.253933:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620918d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22740992->22745087] (req 22740992->22745087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760758.254162:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ad180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484278016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46493us (46712us total) trans 4294972870 rc 0/0 00000100:00100000:2.0:1789760758.254173:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5581 00000100:00100000:2.0:1789760758.254182:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747484785936 time 747538583854 state 0x2 ref 0 00000100:00100000:0.0:1789760758.259680:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760758.259791:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484278144 00000100:00100000:1.0:1789760758.259802:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484278144 00000100:00100000:1.0:1789760758.259852:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747484785936 time 747538583854 state 0x0 ref 1 00000100:00100000:1.0:1789760758.259866:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747538844270 time 747538583854 state 0x1 ref 1 00000100:00100000:1.0:1789760758.259898:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747538844270 time 747590828672 state 0x0 ref 1 00000100:00100000:1.0:1789760758.259910:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484278144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760758.259958:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22745088, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:1.0:1789760758.259967:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.259983:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:1.0:1789760758.259988:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760758.259994:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22745088->22749183] (req 22745088->22749183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.260007:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22745088->22749183] (req 22745088->22749183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760758.310182:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972871 is committed 00002000:00010000:3.0:1789760758.310259:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22745088 -> 22749184 00010000:00010000:3.0:1789760758.310274:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22745088->22749183] (req 22745088->22749183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.310286:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22745088->22749183] (req 22745088->22749183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.310305:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22745088->22749183] (req 22745088->22749183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.310317:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760758.310321:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22745088->22749183] (req 22745088->22749183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.310330:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22745088->22749183] (req 22745088->22749183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.310350:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22745088->22749183] (req 22745088->22749183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.310357:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209194 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22745088->22749183] (req 22745088->22749183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760758.310455:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484278144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50542us (50781us total) trans 4294972871 rc 0/0 00000100:00100000:3.0:1789760758.310465:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5582 00000100:00100000:3.0:1789760758.310475:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747538844270 time 747590828672 state 0x2 ref 0 00000100:00100000:3.0:1789760758.314841:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760758.314939:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484278272 00000100:00100000:1.0:1789760758.314949:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484278272 00000100:00100000:1.0:1789760758.315007:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747538844270 time 747590828672 state 0x0 ref 1 00000100:00100000:1.0:1789760758.315024:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747591089088 time 747590828672 state 0x1 ref 1 00000100:00100000:1.0:1789760758.315059:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747591089088 time 747645989775 state 0x0 ref 1 00000100:00100000:1.0:1789760758.315112:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484278272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760758.315161:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22749184, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:1.0:1789760758.315170:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.315186:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:1.0:1789760758.315204:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760758.315212:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22749184->22753279] (req 22749184->22753279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.315226:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22749184->22753279] (req 22749184->22753279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760758.384722:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972872 is committed 00002000:00010000:3.0:1789760758.384907:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22749184 -> 22753280 00010000:00010000:3.0:1789760758.384930:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22749184->22753279] (req 22749184->22753279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.384946:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22749184->22753279] (req 22749184->22753279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.384964:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22749184->22753279] (req 22749184->22753279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.384972:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760758.384985:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22749184->22753279] (req 22749184->22753279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.384998:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22749184->22753279] (req 22749184->22753279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.385021:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22749184->22753279] (req 22749184->22753279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.385029:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620919b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22749184->22753279] (req 22749184->22753279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760758.385309:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484278272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 70195us (70473us total) trans 4294972872 rc 0/0 00000100:00100000:3.0:1789760758.385319:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5583 00000100:00100000:3.0:1789760758.385328:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747591089088 time 747645989775 state 0x2 ref 0 00000100:00100000:2.0:1789760758.397877:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760758.397998:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484278400 00000100:00100000:3.0:1789760758.398011:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484278400 00000100:00100000:3.0:1789760758.398125:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747591089088 time 747645989775 state 0x0 ref 1 00000100:00100000:3.0:1789760758.398146:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747646250191 time 747645989775 state 0x1 ref 1 00000100:00100000:3.0:1789760758.398204:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747646250191 time 747729133367 state 0x0 ref 1 00000100:00100000:3.0:1789760758.398222:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ac380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484278400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760758.398280:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22753280, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:3.0:1789760758.398290:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.398306:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:3.0:1789760758.398311:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760758.398316:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22753280->22757375] (req 22753280->22757375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.398330:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22753280->22757375] (req 22753280->22757375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760758.456128:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972873 is committed 00002000:00010000:3.0:1789760758.456247:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22753280 -> 22757376 00010000:00010000:3.0:1789760758.456262:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22753280->22757375] (req 22753280->22757375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.456285:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22753280->22757375] (req 22753280->22757375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.456296:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22753280->22757375] (req 22753280->22757375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.456303:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760758.456310:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22753280->22757375] (req 22753280->22757375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.456319:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22753280->22757375] (req 22753280->22757375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.456338:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22753280->22757375] (req 22753280->22757375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.456346:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091a2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22753280->22757375] (req 22753280->22757375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760758.456445:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ac380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484278400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58224us (58573us total) trans 4294972873 rc 0/0 00000100:00100000:3.0:1789760758.456456:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5584 00000100:00100000:3.0:1789760758.457798:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747646250191 time 747729133367 state 0x2 ref 0 00000100:00100000:1.0:1789760758.459441:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760758.459509:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484278528 00000100:00100000:3.0:1789760758.459518:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484278528 00000100:00100000:3.0:1789760758.459564:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747646250191 time 747729133367 state 0x0 ref 1 00000100:00100000:3.0:1789760758.459580:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747729393783 time 747729133367 state 0x1 ref 1 00000100:00100000:3.0:1789760758.459606:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747729393783 time 747790537653 state 0x0 ref 1 00000100:00100000:3.0:1789760758.459619:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484278528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760758.459664:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22757376, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:3.0:1789760758.459674:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.459689:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:3.0:1789760758.459693:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760758.459700:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22757376->22761471] (req 22757376->22761471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.459715:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22757376->22761471] (req 22757376->22761471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760758.509289:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972874 is committed 00002000:00010000:1.0:1789760758.509563:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22757376 -> 22761472 00010000:00010000:1.0:1789760758.509580:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22757376->22761471] (req 22757376->22761471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.509593:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22757376->22761471] (req 22757376->22761471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.509603:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22757376->22761471] (req 22757376->22761471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.509611:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760758.509616:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22757376->22761471] (req 22757376->22761471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.509626:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22757376->22761471] (req 22757376->22761471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.509645:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22757376->22761471] (req 22757376->22761471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.509653:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091a9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22757376->22761471] (req 22757376->22761471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760758.509751:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484278528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50129us (50311us total) trans 4294972874 rc 0/0 00000100:00100000:1.0:1789760758.509762:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5585 00000100:00100000:1.0:1789760758.509775:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747729393783 time 747790537653 state 0x2 ref 0 00000100:00100000:2.0:1789760758.513623:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760758.513789:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484278656 00000100:00100000:1.0:1789760758.513798:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484278656 00000100:00100000:1.0:1789760758.513844:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747729393783 time 747790537653 state 0x0 ref 1 00000100:00100000:1.0:1789760758.513855:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747790798069 time 747790537653 state 0x1 ref 1 00000100:00100000:1.0:1789760758.513879:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747790798069 time 747844809668 state 0x0 ref 1 00000100:00100000:1.0:1789760758.513891:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484278656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760758.513939:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22761472, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:1.0:1789760758.513949:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.513963:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:1.0:1789760758.513967:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760758.513973:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22761472->22765567] (req 22761472->22765567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.513986:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22761472->22765567] (req 22761472->22765567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760758.547522:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972875 is committed 00002000:00010000:1.0:1789760758.547862:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22761472 -> 22765568 00010000:00010000:1.0:1789760758.547878:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22761472->22765567] (req 22761472->22765567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.547890:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22761472->22765567] (req 22761472->22765567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.547899:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22761472->22765567] (req 22761472->22765567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.547906:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760758.547910:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22761472->22765567] (req 22761472->22765567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.547920:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22761472->22765567] (req 22761472->22765567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.547945:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22761472->22765567] (req 22761472->22765567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.547952:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091b0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22761472->22765567] (req 22761472->22765567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760758.548035:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484278656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34141us (34415us total) trans 4294972875 rc 0/0 00000100:00100000:1.0:1789760758.548044:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5586 00000100:00100000:1.0:1789760758.548052:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747790798069 time 747844809668 state 0x2 ref 0 00000100:00100000:2.0:1789760758.555051:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760758.555170:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484278784 00000100:00100000:1.0:1789760758.555179:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484278784 00000100:00100000:0.0:1789760758.555808:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747790798069 time 747844809668 state 0x0 ref 1 00000100:00100000:0.0:1789760758.555823:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747845070084 time 747844809668 state 0x1 ref 1 00000100:00100000:0.0:1789760758.555851:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747845070084 time 747886782024 state 0x0 ref 1 00000100:00100000:0.0:1789760758.555863:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484278784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760758.555907:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22765568, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:0.0:1789760758.555916:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.555932:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:0.0:1789760758.555936:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760758.555942:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22765568->22769663] (req 22765568->22769663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760758.555956:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22765568->22769663] (req 22765568->22769663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760758.601884:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972876 is committed 00002000:00010000:1.0:1789760758.602141:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22765568 -> 22769664 00010000:00010000:1.0:1789760758.602157:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22765568->22769663] (req 22765568->22769663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.602169:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22765568->22769663] (req 22765568->22769663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.602178:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22765568->22769663] (req 22765568->22769663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.602185:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760758.602200:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22765568->22769663] (req 22765568->22769663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.602213:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22765568->22769663] (req 22765568->22769663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.602454:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22765568->22769663] (req 22765568->22769663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.602463:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962091b7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22765568->22769663] (req 22765568->22769663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760758.602594:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484278784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46728us (47548us total) trans 4294972876 rc 0/0 00000100:00100000:1.0:1789760758.602605:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5587 00000100:00100000:1.0:1789760758.602615:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747845070084 time 747886782024 state 0x2 ref 0 00000100:00100000:1.0:1789760758.610668:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760758.610761:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484278912 00000100:00100000:3.0:1789760758.610770:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484278912 00000100:00100000:3.0:1789760758.610826:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747845070084 time 747886782024 state 0x0 ref 1 00000100:00100000:3.0:1789760758.610841:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747887042440 time 747886782024 state 0x1 ref 1 00000100:00100000:3.0:1789760758.610900:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747887042440 time 747941830988 state 0x0 ref 1 00000100:00100000:3.0:1789760758.610913:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484278912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760758.610980:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22769664, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:3.0:1789760758.610990:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.611012:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:3.0:1789760758.611017:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760758.611023:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22769664->22773759] (req 22769664->22773759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.611037:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22769664->22773759] (req 22769664->22773759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760758.644482:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972877 is committed 00002000:00010000:3.0:1789760758.644498:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22769664 -> 22773760 00010000:00010000:3.0:1789760758.644512:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22769664->22773759] (req 22769664->22773759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.644529:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22769664->22773759] (req 22769664->22773759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.644539:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22769664->22773759] (req 22769664->22773759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.644547:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760758.644551:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22769664->22773759] (req 22769664->22773759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.644561:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22769664->22773759] (req 22769664->22773759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.644581:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22769664->22773759] (req 22769664->22773759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.644590:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091be lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22769664->22773759] (req 22769664->22773759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760758.644674:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484278912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33759us (34011us total) trans 4294972877 rc 0/0 00000100:00100000:3.0:1789760758.644683:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5588 00000100:00100000:3.0:1789760758.644693:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747887042440 time 747941830988 state 0x2 ref 0 00000100:00100000:2.0:1789760758.649657:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760758.649717:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484279040 00000100:00100000:1.0:1789760758.649726:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484279040 00000100:00100000:1.0:1789760758.649779:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747887042440 time 747941830988 state 0x0 ref 1 00000100:00100000:1.0:1789760758.649794:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747942091404 time 747941830988 state 0x1 ref 1 00000100:00100000:1.0:1789760758.649825:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747942091404 time 747980756915 state 0x0 ref 1 00000100:00100000:1.0:1789760758.649837:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484279040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760758.649885:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22773760, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:1.0:1789760758.649894:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.649910:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=747s v=(0 0) 00010000:00010000:1.0:1789760758.649914:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760758.649919:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22773760->22777855] (req 22773760->22777855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.649930:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22773760->22777855] (req 22773760->22777855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760758.669679:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508929024:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760758.669752:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00010000:00010000:3.0:1789760758.669869:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760758.669882:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508929024 00000100:00100000:3.0:1789760758.669890:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508929024 00000100:00100000:3.0:1789760758.669934:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 394 00000100:00100000:3.0:1789760758.669944:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba75180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508929024:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:3.0:1789760758.669967:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528023552 pending 1703936 free 3835691008 avail 3816816640 00000100:00100000:2.0:1789760758.670406:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760758.670433:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf501ba0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508929280:0@lo:41:osp-pre-0-1.0 00000100:00100000:2.0:1789760758.670476:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760758.670580:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508929280 00000100:00100000:0.0:1789760758.670586:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508929280 00000100:00100000:0.0:1789760758.670628:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 453 00000100:00100000:0.0:1789760758.670637:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508929280:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:0.0:1789760758.670664:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:0.0:1789760758.670714:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508929280:12345-0@lo:41:osp-pre-0-1.0 Request processed in 78us (237us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760758.670721:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 453 00000100:00100000:0.0:1789760758.670740:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508929152:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760758.670763:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760758.670787:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508929152 00000100:00100000:0.0:1789760758.670791:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508929152 00000100:00100000:1.0:1789760758.670878:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf501ba0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508929280:0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760758.670968:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba75180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508929024:12345-0@lo:13:osp-pre-0-0.0 Request processed in 1024us (1214us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760758.670983:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 394 00000100:00100000:3.0:1789760758.671016:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508929024:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760758.676223:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 395 00000100:00100000:0.0:1789760758.676240:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508929152:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:0.0:1789760758.676267:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528023552 pending 1703936 free 3835691008 avail 3816816640 00000100:00100000:0.0:1789760758.676348:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508929152:12345-0@lo:13:osp-pre-0-1.0 Request processed in 113us (5583us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760758.676356:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 395 00000100:00100000:0.0:1789760758.676411:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508929152:0@lo:13:osp-pre-0-1.0 00000001:00080000:3.0:1789760758.685883:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972878 is committed 00002000:00010000:1.0:1789760758.685995:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22773760 -> 22777856 00010000:00010000:1.0:1789760758.686010:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22773760->22777855] (req 22773760->22777855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.686024:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22773760->22777855] (req 22773760->22777855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.686032:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22773760->22777855] (req 22773760->22777855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.686040:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760758.686044:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22773760->22777855] (req 22773760->22777855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.686053:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22773760->22777855] (req 22773760->22777855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.686533:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22773760->22777855] (req 22773760->22777855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.686545:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091c5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22773760->22777855] (req 22773760->22777855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760758.686646:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484279040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36806us (36995us total) trans 4294972878 rc 0/0 00000100:00100000:1.0:1789760758.686657:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5589 00000100:00100000:1.0:1789760758.686667:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747942091404 time 747980756915 state 0x2 ref 0 00000100:00100000:2.0:1789760758.692562:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760758.695463:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484279168 00000100:00100000:1.0:1789760758.695477:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484279168 00000100:00100000:1.0:1789760758.695540:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747942091404 time 747980756915 state 0x0 ref 1 00000100:00100000:1.0:1789760758.695556:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747981017331 time 747980756915 state 0x1 ref 1 00000100:00100000:1.0:1789760758.695654:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747981017331 time 748026584493 state 0x0 ref 1 00000100:00100000:1.0:1789760758.695668:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484279168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760758.695716:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22777856, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:1.0:1789760758.695725:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.695740:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:1.0:1789760758.695746:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760758.695753:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22777856->22781951] (req 22777856->22781951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.695767:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22777856->22781951] (req 22777856->22781951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760758.732900:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972879 is committed 00002000:00010000:3.0:1789760758.733049:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22777856 -> 22781952 00010000:00010000:3.0:1789760758.733537:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22777856->22781951] (req 22777856->22781951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.733559:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22777856->22781951] (req 22777856->22781951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.733569:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22777856->22781951] (req 22777856->22781951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.733576:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760758.733581:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22777856->22781951] (req 22777856->22781951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.733591:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22777856->22781951] (req 22777856->22781951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.733609:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22777856->22781951] (req 22777856->22781951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.733616:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962091cc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22777856->22781951] (req 22777856->22781951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760758.733814:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484279168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38142us (41256us total) trans 4294972879 rc 0/0 00000100:00100000:2.0:1789760758.733825:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5590 00000100:00100000:2.0:1789760758.733834:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747981017331 time 748026584493 state 0x2 ref 0 00000100:00100000:0.0:1789760758.737506:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760758.737608:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484279296 00000100:00100000:1.0:1789760758.737619:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484279296 00000100:00100000:1.0:1789760758.737672:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 747981017331 time 748026584493 state 0x0 ref 1 00000100:00100000:1.0:1789760758.737688:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748026844909 time 748026584493 state 0x1 ref 1 00000100:00100000:1.0:1789760758.737719:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748026844909 time 748068650248 state 0x0 ref 1 00000100:00100000:1.0:1789760758.737732:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484279296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760758.737779:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22781952, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:1.0:1789760758.737788:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.737803:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:1.0:1789760758.737807:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760758.737812:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22781952->22786047] (req 22781952->22786047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.737825:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22781952->22786047] (req 22781952->22786047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760758.768061:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972880 is committed 00002000:00010000:1.0:1789760758.768321:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22781952 -> 22786048 00010000:00010000:1.0:1789760758.768337:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22781952->22786047] (req 22781952->22786047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.768350:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22781952->22786047] (req 22781952->22786047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.768359:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22781952->22786047] (req 22781952->22786047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.768366:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760758.768371:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22781952->22786047] (req 22781952->22786047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.768380:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22781952->22786047] (req 22781952->22786047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.768399:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22781952->22786047] (req 22781952->22786047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.768406:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091d3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22781952->22786047] (req 22781952->22786047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760758.768537:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484279296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30803us (31038us total) trans 4294972880 rc 0/0 00000100:00100000:1.0:1789760758.768551:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5591 00000100:00100000:1.0:1789760758.768560:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748026844909 time 748068650248 state 0x2 ref 0 00000100:00100000:3.0:1789760758.772928:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760758.776454:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484279424 00000100:00100000:2.0:1789760758.776466:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484279424 00000100:00100000:2.0:1789760758.776531:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748026844909 time 748068650248 state 0x0 ref 1 00000100:00100000:2.0:1789760758.776547:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748068910664 time 748068650248 state 0x1 ref 1 00000100:00100000:2.0:1789760758.776607:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748068910664 time 748107537522 state 0x0 ref 1 00000100:00100000:2.0:1789760758.776622:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484279424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760758.776672:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22786048, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:2.0:1789760758.776681:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.776697:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:2.0:1789760758.776702:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760758.776709:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22786048->22790143] (req 22786048->22790143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.776723:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22786048->22790143] (req 22786048->22790143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760758.807984:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22786048 -> 22790144 00010000:00010000:2.0:1789760758.808005:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22786048->22790143] (req 22786048->22790143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.808018:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22786048->22790143] (req 22786048->22790143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.808027:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22786048->22790143] (req 22786048->22790143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.808034:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760758.808039:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22786048->22790143] (req 22786048->22790143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.808048:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22786048->22790143] (req 22786048->22790143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760758.808132:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972881 is committed 00010000:00010000:2.0:1789760758.808207:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22786048->22790143] (req 22786048->22790143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.808220:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091da lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22786048->22790143] (req 22786048->22790143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760758.808344:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484279424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31721us (35421us total) trans 4294972881 rc 0/0 00000100:00100000:2.0:1789760758.808355:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5592 00000100:00100000:2.0:1789760758.808365:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748068910664 time 748107537522 state 0x2 ref 0 00000100:00100000:2.0:1789760758.811382:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760758.811487:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484279552 00000100:00100000:3.0:1789760758.811498:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484279552 00000100:00100000:3.0:1789760758.811734:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748068910664 time 748107537522 state 0x0 ref 1 00000100:00100000:3.0:1789760758.811751:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748107797938 time 748107537522 state 0x1 ref 1 00000100:00100000:3.0:1789760758.811796:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748107797938 time 748142726884 state 0x0 ref 1 00000100:00100000:3.0:1789760758.811810:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484279552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760758.811862:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22790144, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:3.0:1789760758.811872:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.811888:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:3.0:1789760758.811892:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760758.811899:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22790144->22794239] (req 22790144->22794239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.811913:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22790144->22794239] (req 22790144->22794239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760758.851288:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22790144 -> 22794240 00010000:00010000:1.0:1789760758.851307:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22790144->22794239] (req 22790144->22794239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.851322:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22790144->22794239] (req 22790144->22794239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.851332:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22790144->22794239] (req 22790144->22794239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.851342:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760758.851346:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22790144->22794239] (req 22790144->22794239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.851355:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22790144->22794239] (req 22790144->22794239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.851374:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22790144->22794239] (req 22790144->22794239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.851381:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962091e1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22790144->22794239] (req 22790144->22794239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760758.851484:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484279552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39673us (40107us total) trans 4294972882 rc 0/0 00000100:00100000:1.0:1789760758.851501:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5593 00000100:00100000:1.0:1789760758.851510:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748107797938 time 748142726884 state 0x2 ref 0 00000001:00080000:0.0:1789760758.852092:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972882 is committed 00000100:00100000:0.0:1789760758.856233:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760758.856468:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484279680 00000100:00100000:2.0:1789760758.856479:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484279680 00000100:00100000:2.0:1789760758.856600:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748107797938 time 748142726884 state 0x0 ref 1 00000100:00100000:2.0:1789760758.856615:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748142987300 time 748142726884 state 0x1 ref 1 00000100:00100000:2.0:1789760758.856695:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748142987300 time 748187624981 state 0x0 ref 1 00000100:00100000:2.0:1789760758.856707:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484279680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760758.856755:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22794240, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:2.0:1789760758.856878:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.856896:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:2.0:1789760758.856900:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760758.856906:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22794240->22798335] (req 22794240->22798335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760758.856920:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22794240->22798335] (req 22794240->22798335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760758.888646:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972883 is committed 00002000:00010000:3.0:1789760758.888907:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22794240 -> 22798336 00010000:00010000:3.0:1789760758.888923:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22794240->22798335] (req 22794240->22798335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.888937:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22794240->22798335] (req 22794240->22798335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.889011:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22794240->22798335] (req 22794240->22798335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.889020:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760758.889025:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22794240->22798335] (req 22794240->22798335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.889035:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22794240->22798335] (req 22794240->22798335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.889057:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22794240->22798335] (req 22794240->22798335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.889213:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962091e8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22794240->22798335] (req 22794240->22798335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760758.889318:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484279680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32608us (33092us total) trans 4294972883 rc 0/0 00000100:00100000:3.0:1789760758.889329:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5594 00000100:00100000:3.0:1789760758.889339:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748142987300 time 748187624981 state 0x2 ref 0 00000100:00100000:3.0:1789760758.896038:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760758.896845:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484279808 00000100:00100000:1.0:1789760758.896857:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484279808 00000100:00100000:1.0:1789760758.896916:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748142987300 time 748187624981 state 0x0 ref 1 00000100:00100000:1.0:1789760758.896933:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748187885397 time 748187624981 state 0x1 ref 1 00000100:00100000:1.0:1789760758.896978:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748187885397 time 748227908900 state 0x0 ref 1 00000100:00100000:1.0:1789760758.896991:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba77100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484279808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760758.897043:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22798336, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:1.0:1789760758.897051:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.897084:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:1.0:1789760758.897088:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760758.897094:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22798336->22802431] (req 22798336->22802431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760758.897840:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22798336->22802431] (req 22798336->22802431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760758.966628:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972884 is committed 00002000:00010000:3.0:1789760758.966754:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22798336 -> 22802432 00010000:00010000:3.0:1789760758.966771:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22798336->22802431] (req 22798336->22802431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.966784:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22798336->22802431] (req 22798336->22802431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.966797:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22798336->22802431] (req 22798336->22802431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.966809:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760758.966817:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22798336->22802431] (req 22798336->22802431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.966827:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22798336->22802431] (req 22798336->22802431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.966847:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22798336->22802431] (req 22798336->22802431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.966855:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962091ef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22798336->22802431] (req 22798336->22802431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760758.966943:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba77100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484279808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69949us (70910us total) trans 4294972884 rc 0/0 00000100:00100000:3.0:1789760758.966952:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5595 00000100:00100000:3.0:1789760758.966962:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748187885397 time 748227908900 state 0x2 ref 0 00000100:00100000:2.0:1789760758.971640:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760758.971834:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484279936 00000100:00100000:3.0:1789760758.971844:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484279936 00000100:00100000:3.0:1789760758.971899:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748187885397 time 748227908900 state 0x0 ref 1 00000100:00100000:3.0:1789760758.971914:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748228169316 time 748227908900 state 0x1 ref 1 00000100:00100000:3.0:1789760758.971981:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748228169316 time 748302911039 state 0x0 ref 1 00000100:00100000:3.0:1789760758.971997:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484279936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760758.972046:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22802432, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:3.0:1789760758.972056:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.972092:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:3.0:1789760758.972096:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760758.972103:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22802432->22806527] (req 22802432->22806527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760758.972116:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22802432->22806527] (req 22802432->22806527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760759.046901:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22802432 -> 22806528 00010000:00010000:3.0:1789760759.046936:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22802432->22806527] (req 22802432->22806527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.046951:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22802432->22806527] (req 22802432->22806527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.046959:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22802432->22806527] (req 22802432->22806527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.046976:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760759.046980:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22802432->22806527] (req 22802432->22806527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.046990:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22802432->22806527] (req 22802432->22806527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.047012:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22802432->22806527] (req 22802432->22806527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.047019:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962091f6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22802432->22806527] (req 22802432->22806527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760759.047155:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484279936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 75154us (75519us total) trans 4294972885 rc 0/0 00000100:00100000:3.0:1789760759.047166:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5596 00000100:00100000:3.0:1789760759.047176:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748228169316 time 748302911039 state 0x2 ref 0 00000100:00100000:3.0:1789760759.051433:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760759.051470:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484280064 00000100:00100000:3.0:1789760759.051478:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484280064 00000100:00100000:3.0:1789760759.051528:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748228169316 time 748302911039 state 0x0 ref 1 00000100:00100000:3.0:1789760759.051544:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748303171455 time 748302911039 state 0x1 ref 1 00000100:00100000:3.0:1789760759.051566:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748303171455 time 748382497319 state 0x0 ref 1 00000100:00100000:3.0:1789760759.051577:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484280064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760759.051625:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22806528, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:3.0:1789760759.051635:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.051651:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:3.0:1789760759.051655:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760759.051661:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22806528->22810623] (req 22806528->22810623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.051674:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22806528->22810623] (req 22806528->22810623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760759.052181:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972885 is committed 00000001:00080000:0.0:1789760759.109835:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972886 is committed 00002000:00010000:0.0:1789760759.110438:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22806528 -> 22810624 00010000:00010000:0.0:1789760759.110455:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22806528->22810623] (req 22806528->22810623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.110473:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22806528->22810623] (req 22806528->22810623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.110482:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22806528->22810623] (req 22806528->22810623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.110490:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760759.110495:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22806528->22810623] (req 22806528->22810623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.110503:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22806528->22810623] (req 22806528->22810623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.110525:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22806528->22810623] (req 22806528->22810623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.110533:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962091fd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22806528->22810623] (req 22806528->22810623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760759.110651:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484280064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59069us (59221us total) trans 4294972886 rc 0/0 00000100:00100000:0.0:1789760759.110663:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5597 00000100:00100000:0.0:1789760759.110804:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748303171455 time 748382497319 state 0x2 ref 0 00000100:00100000:3.0:1789760759.115422:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760759.115524:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484280192 00000100:00100000:2.0:1789760759.115534:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484280192 00000100:00100000:2.0:1789760759.115592:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748303171455 time 748382497319 state 0x0 ref 1 00000100:00100000:2.0:1789760759.115607:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748382757735 time 748382497319 state 0x1 ref 1 00000100:00100000:2.0:1789760759.115639:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748382757735 time 748446570512 state 0x0 ref 1 00000100:00100000:2.0:1789760759.115652:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484280192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760759.115699:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22810624, mtime: 0x6aad94f6, atime: 0x0, ctime: 0x6aad94f6, blocks: 0xb007 00010000:00010000:2.0:1789760759.115708:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.115725:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:2.0:1789760759.115729:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760759.115736:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22810624->22814719] (req 22810624->22814719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.115750:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22810624->22814719] (req 22810624->22814719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760759.180268:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba74700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508929408:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760759.180354:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760759.180400:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508929408 00000100:00100000:3.0:1789760759.180409:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508929408 00000100:00100000:3.0:1789760759.180458:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 454 00000100:00100000:3.0:1789760759.180468:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a33480 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508929408:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:3.0:1789760759.180504:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:3.0:1789760759.180553:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a33480 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508929408:12345-0@lo:41:osp-pre-1-0.0 Request processed in 86us (199us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760759.180560:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 454 00000100:00100000:1.0:1789760759.180631:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba74700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508929408:0@lo:41:osp-pre-1-0.0 00000001:00080000:0.0:1789760759.205960:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972887 is committed 00002000:00010000:3.0:1789760759.206355:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22810624 -> 22814720 00002000:00010000:3.0:1789760759.206359:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760758 -> 1789760759 00002000:00010000:3.0:1789760759.206362:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760758 -> 1789760759 00010000:00010000:3.0:1789760759.206376:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22810624->22814719] (req 22810624->22814719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.206389:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22810624->22814719] (req 22810624->22814719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.206399:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22810624->22814719] (req 22810624->22814719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.206407:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760759.206413:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22810624->22814719] (req 22810624->22814719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.206422:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22810624->22814719] (req 22810624->22814719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.206443:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22810624->22814719] (req 22810624->22814719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.206451:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209204 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22810624->22814719] (req 22810624->22814719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760759.206547:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484280192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 90892us (91131us total) trans 4294972887 rc 0/0 00000100:00100000:3.0:1789760759.206558:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5598 00000100:00100000:3.0:1789760759.206567:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748382757735 time 748446570512 state 0x2 ref 0 00000100:00100000:2.0:1789760759.210437:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760759.210524:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484280320 00000100:00100000:3.0:1789760759.210534:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484280320 00000100:00100000:3.0:1789760759.210591:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748382757735 time 748446570512 state 0x0 ref 1 00000100:00100000:3.0:1789760759.210605:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748446830928 time 748446570512 state 0x1 ref 1 00000100:00100000:3.0:1789760759.210642:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748446830928 time 748541572561 state 0x0 ref 1 00000100:00100000:3.0:1789760759.210655:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484280320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760759.210707:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22814720, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:3.0:1789760759.210721:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.210736:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:3.0:1789760759.210740:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760759.210746:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22814720->22818815] (req 22814720->22818815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.210764:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22814720->22818815] (req 22814720->22818815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760759.299496:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972888 is committed 00002000:00010000:2.0:1789760759.299775:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22814720 -> 22818816 00010000:00010000:2.0:1789760759.299791:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22814720->22818815] (req 22814720->22818815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.299804:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22814720->22818815] (req 22814720->22818815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.299813:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22814720->22818815] (req 22814720->22818815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.299825:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760759.299830:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22814720->22818815] (req 22814720->22818815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.299840:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22814720->22818815] (req 22814720->22818815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.299859:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22814720->22818815] (req 22814720->22818815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.299867:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620920b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22814720->22818815] (req 22814720->22818815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760759.299974:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484280320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 89317us (89540us total) trans 4294972888 rc 0/0 00000100:00100000:2.0:1789760759.299984:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5599 00000100:00100000:2.0:1789760759.299994:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748446830928 time 748541572561 state 0x2 ref 0 00000100:00100000:2.0:1789760759.307896:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760759.308044:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484280448 00000100:00100000:3.0:1789760759.308055:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484280448 00000100:00100000:3.0:1789760759.308151:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748446830928 time 748541572561 state 0x0 ref 1 00000100:00100000:3.0:1789760759.308168:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748541832977 time 748541572561 state 0x1 ref 1 00000100:00100000:3.0:1789760759.308203:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748541832977 time 748639134051 state 0x0 ref 1 00000100:00100000:3.0:1789760759.308215:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484280448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760759.308263:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22818816, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:3.0:1789760759.308272:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.308288:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:3.0:1789760759.308292:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760759.308299:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22818816->22822911] (req 22818816->22822911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.308313:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22818816->22822911] (req 22818816->22822911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760759.398584:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22818816 -> 22822912 00010000:00010000:3.0:1789760759.398604:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22818816->22822911] (req 22818816->22822911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.398618:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22818816->22822911] (req 22818816->22822911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.398627:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22818816->22822911] (req 22818816->22822911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.398634:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760759.398639:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22818816->22822911] (req 22818816->22822911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.398649:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22818816->22822911] (req 22818816->22822911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.398675:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22818816->22822911] (req 22818816->22822911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.398682:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209212 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22818816->22822911] (req 22818816->22822911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760759.398784:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484280448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 90565us (90891us total) trans 4294972889 rc 0/0 00000100:00100000:3.0:1789760759.398796:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5600 00000100:00100000:3.0:1789760759.398806:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748541832977 time 748639134051 state 0x2 ref 0 00000001:00080000:2.0:1789760759.399407:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972889 is committed 00000100:00100000:3.0:1789760759.401357:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760759.401452:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484280576 00000100:00100000:0.0:1789760759.401462:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484280576 00000100:00100000:0.0:1789760759.401522:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748541832977 time 748639134051 state 0x0 ref 1 00000100:00100000:0.0:1789760759.401538:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748639394467 time 748639134051 state 0x1 ref 1 00000100:00100000:0.0:1789760759.401620:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748639394467 time 748732550376 state 0x0 ref 1 00000100:00100000:0.0:1789760759.401635:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484280576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760759.401682:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22822912, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:0.0:1789760759.401691:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.401707:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:0.0:1789760759.401711:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760759.401718:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22822912->22827007] (req 22822912->22827007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.401731:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22822912->22827007] (req 22822912->22827007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760759.430508:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22822912 -> 22827008 00010000:00010000:2.0:1789760759.430527:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22822912->22827007] (req 22822912->22827007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.430540:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22822912->22827007] (req 22822912->22827007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.430550:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22822912->22827007] (req 22822912->22827007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.430558:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760759.430563:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22822912->22827007] (req 22822912->22827007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.430572:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22822912->22827007] (req 22822912->22827007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.430591:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22822912->22827007] (req 22822912->22827007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.430598:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209219 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22822912->22827007] (req 22822912->22827007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760759.430692:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484280576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29057us (29340us total) trans 4294972890 rc 0/0 00000100:00100000:2.0:1789760759.430702:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5601 00000100:00100000:2.0:1789760759.430711:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748639394467 time 748732550376 state 0x2 ref 0 00000001:00080000:2.0:1789760759.430749:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972890 is committed 00000100:00100000:1.0:1789760759.438723:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760759.441204:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484280704 00000100:00100000:2.0:1789760759.441288:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484280704 00000100:00100000:3.0:1789760759.443296:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748639394467 time 748732550376 state 0x0 ref 1 00000100:00100000:3.0:1789760759.443316:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748732810792 time 748732550376 state 0x1 ref 1 00000100:00100000:3.0:1789760759.443413:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748732810792 time 748774343307 state 0x0 ref 1 00000100:00100000:3.0:1789760759.443428:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a32d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484280704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760759.443477:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22827008, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:3.0:1789760759.443486:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.443500:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:3.0:1789760759.443505:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760759.443512:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22827008->22831103] (req 22827008->22831103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.443525:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22827008->22831103] (req 22827008->22831103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760759.513557:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972891 is committed 00002000:00010000:0.0:1789760759.516250:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22827008 -> 22831104 00010000:00010000:0.0:1789760759.516268:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22827008->22831103] (req 22827008->22831103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.516287:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22827008->22831103] (req 22827008->22831103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.516296:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22827008->22831103] (req 22827008->22831103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.516303:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760759.516307:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22827008->22831103] (req 22827008->22831103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.516316:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22827008->22831103] (req 22827008->22831103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.516334:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22827008->22831103] (req 22827008->22831103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.516342:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209220 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22827008->22831103] (req 22827008->22831103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760759.516438:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a32d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484280704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73009us (77719us total) trans 4294972891 rc 0/0 00000100:00100000:0.0:1789760759.516448:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5602 00000100:00100000:0.0:1789760759.516458:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748732810792 time 748774343307 state 0x2 ref 0 00000100:00100000:1.0:1789760759.522056:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760759.522309:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484280832 00000100:00100000:2.0:1789760759.522319:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484280832 00000100:00100000:2.0:1789760759.522377:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748732810792 time 748774343307 state 0x0 ref 1 00000100:00100000:2.0:1789760759.522392:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748774603723 time 748774343307 state 0x1 ref 1 00000100:00100000:2.0:1789760759.522462:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748774603723 time 748853392208 state 0x0 ref 1 00000100:00100000:2.0:1789760759.522475:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a33800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484280832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760759.522519:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22831104, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:2.0:1789760759.522528:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.522541:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:2.0:1789760759.522545:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760759.522551:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22831104->22835199] (req 22831104->22835199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.522565:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22831104->22835199] (req 22831104->22835199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760759.571004:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22831104 -> 22835200 00010000:00010000:2.0:1789760759.571024:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22831104->22835199] (req 22831104->22835199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.571037:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22831104->22835199] (req 22831104->22835199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.571046:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22831104->22835199] (req 22831104->22835199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.571054:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760759.571058:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22831104->22835199] (req 22831104->22835199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.571108:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22831104->22835199] (req 22831104->22835199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.571128:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22831104->22835199] (req 22831104->22835199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.571136:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209227 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22831104->22835199] (req 22831104->22835199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760759.571229:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a33800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484280832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48751us (49178us total) trans 4294972892 rc 0/0 00000100:00100000:2.0:1789760759.571241:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5603 00000100:00100000:2.0:1789760759.571249:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748774603723 time 748853392208 state 0x2 ref 0 00000001:00080000:0.0:1789760759.571745:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972892 is committed 00000100:00100000:0.0:1789760759.577635:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760759.578226:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484280960 00000100:00100000:2.0:1789760759.578238:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484280960 00000100:00100000:2.0:1789760759.578290:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748774603723 time 748853392208 state 0x0 ref 1 00000100:00100000:2.0:1789760759.578306:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748853652624 time 748853392208 state 0x1 ref 1 00000100:00100000:2.0:1789760759.578325:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748853652624 time 748909255964 state 0x0 ref 1 00000100:00100000:2.0:1789760759.578340:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484280960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760759.578389:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22835200, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:2.0:1789760759.578397:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.578478:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:2.0:1789760759.578484:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760759.578490:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22835200->22839295] (req 22835200->22839295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.578504:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22835200->22839295] (req 22835200->22839295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760759.635634:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22835200 -> 22839296 00010000:00010000:0.0:1789760759.635665:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22835200->22839295] (req 22835200->22839295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.635681:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22835200->22839295] (req 22835200->22839295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.635691:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22835200->22839295] (req 22835200->22839295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.635699:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760759.635705:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22835200->22839295] (req 22835200->22839295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.635716:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22835200->22839295] (req 22835200->22839295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.635743:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22835200->22839295] (req 22835200->22839295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.635752:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620922e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22835200->22839295] (req 22835200->22839295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760759.635902:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484280960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57555us (58267us total) trans 4294972893 rc 0/0 00000100:00100000:0.0:1789760759.635917:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5604 00000100:00100000:0.0:1789760759.635933:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748853652624 time 748909255964 state 0x2 ref 0 00000001:00080000:1.0:1789760759.636546:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972893 is committed 00000100:00100000:1.0:1789760759.642234:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760759.642422:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484281088 00000100:00100000:0.0:1789760759.642435:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484281088 00000100:00100000:0.0:1789760759.642516:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748853652624 time 748909255964 state 0x0 ref 1 00000100:00100000:0.0:1789760759.642537:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748909516380 time 748909255964 state 0x1 ref 1 00000100:00100000:0.0:1789760759.642631:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748909516380 time 748973560681 state 0x0 ref 1 00000100:00100000:0.0:1789760759.642648:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a32680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484281088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760759.642712:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22839296, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:0.0:1789760759.642723:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.642743:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=748s v=(0 0) 00010000:00010000:0.0:1789760759.642749:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760759.642756:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22839296->22843391] (req 22839296->22843391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.642790:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22839296->22843391] (req 22839296->22843391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.695499:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760759.712419:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972894 is committed 00002000:00010000:0.0:1789760759.712550:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22839296 -> 22843392 00010000:00010000:0.0:1789760759.712572:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22839296->22843391] (req 22839296->22843391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.712593:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22839296->22843391] (req 22839296->22843391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.712603:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22839296->22843391] (req 22839296->22843391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.712612:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760759.712617:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22839296->22843391] (req 22839296->22843391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.712628:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22839296->22843391] (req 22839296->22843391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.712653:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22839296->22843391] (req 22839296->22843391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.712728:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209235 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22839296->22843391] (req 22839296->22843391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760759.712967:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a32680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484281088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 70310us (70734us total) trans 4294972894 rc 0/0 00000100:00100000:0.0:1789760759.712981:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5605 00000100:00100000:0.0:1789760759.712995:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748909516380 time 748973560681 state 0x2 ref 0 00000100:00100000:3.0:1789760759.723166:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760759.723253:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484281216 00000100:00100000:1.0:1789760759.723267:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484281216 00000100:00100000:1.0:1789760759.723558:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748909516380 time 748973560681 state 0x0 ref 1 00000100:00100000:1.0:1789760759.723581:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748973821097 time 748973560681 state 0x1 ref 1 00000100:00100000:1.0:1789760759.723689:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748973821097 time 749054617450 state 0x0 ref 1 00000100:00100000:1.0:1789760759.723706:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484281216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760759.723785:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22843392, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:1.0:1789760759.723796:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.723816:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:1.0:1789760759.723822:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760759.723831:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22843392->22847487] (req 22843392->22847487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.725726:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22843392->22847487] (req 22843392->22847487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760759.777503:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22843392 -> 22847488 00010000:00010000:1.0:1789760759.777532:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22843392->22847487] (req 22843392->22847487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.777547:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22843392->22847487] (req 22843392->22847487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.777557:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22843392->22847487] (req 22843392->22847487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.777565:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760759.777570:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22843392->22847487] (req 22843392->22847487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.777580:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22843392->22847487] (req 22843392->22847487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.777607:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22843392->22847487] (req 22843392->22847487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.777616:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620923c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22843392->22847487] (req 22843392->22847487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760759.777755:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484281216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54047us (54607us total) trans 4294972895 rc 0/0 00000100:00100000:1.0:1789760759.777768:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5606 00000100:00100000:1.0:1789760759.777780:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748973821097 time 749054617450 state 0x2 ref 0 00000001:00080000:3.0:1789760759.778647:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972895 is committed 00000100:00100000:0.0:1789760759.782919:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760759.783128:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484281344 00000100:00100000:2.0:1789760759.783142:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484281344 00000100:00100000:2.0:1789760759.783229:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 748973821097 time 749054617450 state 0x0 ref 1 00000100:00100000:2.0:1789760759.783248:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749054877866 time 749054617450 state 0x1 ref 1 00000100:00100000:2.0:1789760759.783318:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749054877866 time 749114248104 state 0x0 ref 1 00000100:00100000:2.0:1789760759.783334:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484281344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760759.783413:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22847488, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:2.0:1789760759.783426:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.783447:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:2.0:1789760759.783453:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760759.783460:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22847488->22851583] (req 22847488->22851583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.783477:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22847488->22851583] (req 22847488->22851583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760759.819050:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22847488 -> 22851584 00010000:00010000:3.0:1789760759.819111:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22847488->22851583] (req 22847488->22851583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.819128:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22847488->22851583] (req 22847488->22851583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.819138:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22847488->22851583] (req 22847488->22851583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.819146:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760759.819151:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22847488->22851583] (req 22847488->22851583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.819161:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22847488->22851583] (req 22847488->22851583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.819196:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22847488->22851583] (req 22847488->22851583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.819206:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209243 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22847488->22851583] (req 22847488->22851583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760759.819312:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484281344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35974us (36397us total) trans 4294972896 rc 0/0 00000100:00100000:3.0:1789760759.819323:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5607 00000100:00100000:3.0:1789760759.819334:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749054877866 time 749114248104 state 0x2 ref 0 00000100:00100000:1.0:1789760759.823022:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760759.823404:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484281472 00000001:00080000:0.0:1789760759.823411:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972896 is committed 00000100:00100000:2.0:1789760759.823419:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484281472 00000100:00100000:2.0:1789760759.823507:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749054877866 time 749114248104 state 0x0 ref 1 00000100:00100000:2.0:1789760759.823526:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749114508520 time 749114248104 state 0x1 ref 1 00000100:00100000:2.0:1789760759.823562:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749114508520 time 749154492671 state 0x0 ref 1 00000100:00100000:2.0:1789760759.823576:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484281472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760759.823640:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22851584, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:2.0:1789760759.823650:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.823681:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:2.0:1789760759.823686:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760759.823694:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22851584->22855679] (req 22851584->22855679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.823717:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22851584->22855679] (req 22851584->22855679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760759.865114:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972897 is committed 00002000:00010000:3.0:1789760759.865384:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22851584 -> 22855680 00010000:00010000:3.0:1789760759.865410:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22851584->22855679] (req 22851584->22855679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.865428:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22851584->22855679] (req 22851584->22855679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.865438:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22851584->22855679] (req 22851584->22855679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.865447:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760759.865453:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22851584->22855679] (req 22851584->22855679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.865464:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22851584->22855679] (req 22851584->22855679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.865495:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22851584->22855679] (req 22851584->22855679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.865504:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620924a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22851584->22855679] (req 22851584->22855679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760759.865640:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484281472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42059us (42627us total) trans 4294972897 rc 0/0 00000100:00100000:3.0:1789760759.865651:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5608 00000100:00100000:3.0:1789760759.865663:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749114508520 time 749154492671 state 0x2 ref 0 00000100:00100000:2.0:1789760759.873025:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760759.873100:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484281600 00000100:00100000:0.0:1789760759.873114:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484281600 00000100:00100000:0.0:1789760759.873210:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749114508520 time 749154492671 state 0x0 ref 1 00000100:00100000:0.0:1789760759.873232:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749154753087 time 749154492671 state 0x1 ref 1 00000100:00100000:0.0:1789760759.873273:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749154753087 time 749204195997 state 0x0 ref 1 00000100:00100000:0.0:1789760759.873290:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484281600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760759.873363:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22855680, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:0.0:1789760759.873375:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.873396:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:0.0:1789760759.873401:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760759.873409:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22855680->22859775] (req 22855680->22859775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760759.873425:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22855680->22859775] (req 22855680->22859775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760759.949581:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972898 is committed 00002000:00010000:1.0:1789760759.949741:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22855680 -> 22859776 00010000:00010000:1.0:1789760759.949761:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22855680->22859775] (req 22855680->22859775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.949777:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22855680->22859775] (req 22855680->22859775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.949789:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22855680->22859775] (req 22855680->22859775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.949797:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760759.949803:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22855680->22859775] (req 22855680->22859775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.949813:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22855680->22859775] (req 22855680->22859775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.949839:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22855680->22859775] (req 22855680->22859775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.949853:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209251 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22855680->22859775] (req 22855680->22859775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760759.950157:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484281600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 76863us (77139us total) trans 4294972898 rc 0/0 00000100:00100000:1.0:1789760759.950169:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5609 00000100:00100000:1.0:1789760759.950191:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749154753087 time 749204195997 state 0x2 ref 0 00000100:00100000:0.0:1789760759.954475:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760759.954605:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484281728 00000100:00100000:1.0:1789760759.954618:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484281728 00000100:00100000:1.0:1789760759.954695:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749154753087 time 749204195997 state 0x0 ref 1 00000100:00100000:1.0:1789760759.954715:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749204456413 time 749204195997 state 0x1 ref 1 00000100:00100000:1.0:1789760759.954755:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749204456413 time 749285685241 state 0x0 ref 1 00000100:00100000:1.0:1789760759.954770:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484281728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760759.954841:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22859776, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:1.0:1789760759.954853:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.954874:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:1.0:1789760759.954879:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760759.954886:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22859776->22863871] (req 22859776->22863871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760759.954907:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22859776->22863871] (req 22859776->22863871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760759.989107:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972899 is committed 00002000:00010000:2.0:1789760759.989395:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22859776 -> 22863872 00010000:00010000:2.0:1789760759.989415:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22859776->22863871] (req 22859776->22863871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.989431:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22859776->22863871] (req 22859776->22863871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.989443:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22859776->22863871] (req 22859776->22863871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.989452:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760759.989458:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22859776->22863871] (req 22859776->22863871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.989470:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22859776->22863871] (req 22859776->22863871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.989496:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22859776->22863871] (req 22859776->22863871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760759.989505:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209258 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22859776->22863871] (req 22859776->22863871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760759.989640:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484281728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34864us (35173us total) trans 4294972899 rc 0/0 00000100:00100000:2.0:1789760759.989653:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5610 00000100:00100000:2.0:1789760759.989667:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749204456413 time 749285685241 state 0x2 ref 0 00000100:00100000:1.0:1789760759.993528:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760759.993797:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484281856 00000100:00100000:3.0:1789760759.993810:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484281856 00000100:00100000:3.0:1789760759.993888:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749204456413 time 749285685241 state 0x0 ref 1 00000100:00100000:3.0:1789760759.993906:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749285945657 time 749285685241 state 0x1 ref 1 00000100:00100000:3.0:1789760759.994029:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749285945657 time 749324958689 state 0x0 ref 1 00000100:00100000:3.0:1789760759.994045:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706d450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484281856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760759.994274:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22863872, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:3.0:1789760759.994285:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.994312:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:3.0:1789760759.994317:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760759.994325:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22863872->22867967] (req 22863872->22867967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760759.994341:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22863872->22867967] (req 22863872->22867967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760760.037246:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972900 is committed 00002000:00010000:0.0:1789760760.037383:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22863872 -> 22867968 00010000:00010000:0.0:1789760760.037405:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22863872->22867967] (req 22863872->22867967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.037420:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22863872->22867967] (req 22863872->22867967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.037430:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22863872->22867967] (req 22863872->22867967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.037439:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.037444:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22863872->22867967] (req 22863872->22867967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.037456:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22863872->22867967] (req 22863872->22867967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.037480:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22863872->22867967] (req 22863872->22867967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.037488:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620925f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22863872->22867967] (req 22863872->22867967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.037617:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706d450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484281856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43570us (44099us total) trans 4294972900 rc 0/0 00000100:00100000:0.0:1789760760.037629:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5611 00000100:00100000:0.0:1789760760.037641:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749285945657 time 749324958689 state 0x2 ref 0 00000100:00100000:3.0:1789760760.041818:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760760.041925:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484281984 00000100:00100000:0.0:1789760760.041938:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484281984 00000100:00080000:2.0:1789760760.042014:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760760.042032:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749285945657 time 749324958689 state 0x0 ref 1 00000100:00100000:0.0:1789760760.042063:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749325219105 time 749324958689 state 0x1 ref 1 00000100:00100000:0.0:1789760760.042146:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749325219105 time 749373075934 state 0x0 ref 1 00000100:00100000:0.0:1789760760.042161:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484281984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760760.042239:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22867968, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:0.0:1789760760.042250:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.042270:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:0.0:1789760760.042275:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760760.042283:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22867968->22872063] (req 22867968->22872063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.042299:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22867968->22872063] (req 22867968->22872063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760760.072531:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22867968 -> 22872064 00010000:00010000:1.0:1789760760.072559:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22867968->22872063] (req 22867968->22872063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.072575:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22867968->22872063] (req 22867968->22872063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.072586:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22867968->22872063] (req 22867968->22872063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.072596:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760760.072602:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22867968->22872063] (req 22867968->22872063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.072614:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22867968->22872063] (req 22867968->22872063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.072645:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22867968->22872063] (req 22867968->22872063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.072655:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209266 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22867968->22872063] (req 22867968->22872063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760760.072671:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972901 is committed 00000100:00100000:1.0:1789760760.072931:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484281984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30767us (31124us total) trans 4294972901 rc 0/0 00000100:00100000:1.0:1789760760.072943:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5612 00000100:00100000:1.0:1789760760.072956:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749325219105 time 749373075934 state 0x2 ref 0 00000100:00100000:2.0:1789760760.080436:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760760.080647:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484282112 00000100:00100000:1.0:1789760760.080725:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484282112 00000100:00100000:1.0:1789760760.080862:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749325219105 time 749373075934 state 0x0 ref 1 00000100:00100000:1.0:1789760760.080883:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749373336350 time 749373075934 state 0x1 ref 1 00000100:00100000:1.0:1789760760.080970:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749373336350 time 749411891788 state 0x0 ref 1 00000100:00100000:1.0:1789760760.080987:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484282112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760760.081052:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22872064, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:1.0:1789760760.081097:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.081119:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:1.0:1789760760.081124:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760760.081133:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22872064->22876159] (req 22872064->22876159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.081153:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22872064->22876159] (req 22872064->22876159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760760.120674:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972902 is committed 00002000:00010000:0.0:1789760760.120804:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22872064 -> 22876160 00010000:00010000:0.0:1789760760.120826:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22872064->22876159] (req 22872064->22876159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.120842:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22872064->22876159] (req 22872064->22876159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.120858:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22872064->22876159] (req 22872064->22876159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.120865:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.120870:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22872064->22876159] (req 22872064->22876159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.120880:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22872064->22876159] (req 22872064->22876159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.120905:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22872064->22876159] (req 22872064->22876159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.120913:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620926d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22872064->22876159] (req 22872064->22876159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760760.122549:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484282112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41551us (42113us total) trans 4294972902 rc 0/0 00000100:00100000:1.0:1789760760.122565:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5613 00000100:00100000:1.0:1789760760.122578:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749373336350 time 749411891788 state 0x2 ref 0 00000100:00100000:2.0:1789760760.130555:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760760.130728:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484282240 00000100:00100000:1.0:1789760760.130742:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484282240 00000100:00100000:1.0:1789760760.130817:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749373336350 time 749411891788 state 0x0 ref 1 00000100:00100000:1.0:1789760760.130833:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749412152204 time 749411891788 state 0x1 ref 1 00000100:00100000:0.0:1789760760.130924:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749412152204 time 749461852175 state 0x0 ref 1 00000100:00100000:0.0:1789760760.130950:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760760.131131:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22876160, mtime: 0x6aad94f7, atime: 0x0, ctime: 0x6aad94f7, blocks: 0xb007 00010000:00010000:0.0:1789760760.131143:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.131164:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:0.0:1789760760.131170:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760760.131187:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22876160->22880255] (req 22876160->22880255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.131206:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22876160->22880255] (req 22876160->22880255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760760.173277:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972903 is committed 00002000:00010000:0.0:1789760760.173377:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22876160 -> 22880256 00002000:00010000:0.0:1789760760.173383:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760759 -> 1789760760 00002000:00010000:0.0:1789760760.173386:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760759 -> 1789760760 00010000:00010000:0.0:1789760760.173402:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22876160->22880255] (req 22876160->22880255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.173414:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22876160->22880255] (req 22876160->22880255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.173421:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22876160->22880255] (req 22876160->22880255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.173426:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.173431:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22876160->22880255] (req 22876160->22880255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.173438:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22876160->22880255] (req 22876160->22880255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.173457:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22876160->22880255] (req 22876160->22880255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.173463:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209274 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22876160->22880255] (req 22876160->22880255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.173570:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42626us (43028us total) trans 4294972903 rc 0/0 00000100:00100000:0.0:1789760760.173581:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5614 00000100:00100000:0.0:1789760760.173593:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749412152204 time 749461852175 state 0x2 ref 0 00000100:00100000:3.0:1789760760.176430:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760760.176536:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484282368 00000100:00100000:0.0:1789760760.176548:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484282368 00000100:00100000:0.0:1789760760.176633:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749412152204 time 749461852175 state 0x0 ref 1 00000100:00100000:0.0:1789760760.176654:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749462112591 time 749461852175 state 0x1 ref 1 00000100:00100000:0.0:1789760760.176749:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749462112591 time 749507677851 state 0x0 ref 1 00000100:00100000:0.0:1789760760.176768:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484282368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760760.176841:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22880256, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:0.0:1789760760.176852:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.176874:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:0.0:1789760760.176879:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760760.176887:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22880256->22884351] (req 22880256->22884351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.176911:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22880256->22884351] (req 22880256->22884351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760760.213162:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972904 is committed 00002000:00010000:0.0:1789760760.213186:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22880256 -> 22884352 00010000:00010000:0.0:1789760760.213205:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22880256->22884351] (req 22880256->22884351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.213222:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22880256->22884351] (req 22880256->22884351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.213233:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22880256->22884351] (req 22880256->22884351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.213242:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.213248:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22880256->22884351] (req 22880256->22884351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.213259:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22880256->22884351] (req 22880256->22884351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.213286:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22880256->22884351] (req 22880256->22884351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.213295:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620927b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22880256->22884351] (req 22880256->22884351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.213431:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484282368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36662us (37007us total) trans 4294972904 rc 0/0 00000100:00100000:0.0:1789760760.213444:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5615 00000100:00100000:0.0:1789760760.213457:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749462112591 time 749507677851 state 0x2 ref 0 00000100:00100000:2.0:1789760760.216122:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760760.216295:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484282496 00000100:00100000:1.0:1789760760.216308:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484282496 00000100:00100000:1.0:1789760760.216384:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749462112591 time 749507677851 state 0x0 ref 1 00000100:00100000:1.0:1789760760.216403:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749507938267 time 749507677851 state 0x1 ref 1 00000100:00100000:1.0:1789760760.216440:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749507938267 time 749547370182 state 0x0 ref 1 00000100:00100000:1.0:1789760760.216455:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760760.216526:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22884352, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:1.0:1789760760.216536:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.216561:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:1.0:1789760760.216565:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760760.216570:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22884352->22888447] (req 22884352->22888447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.216583:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22884352->22888447] (req 22884352->22888447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760760.243099:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972905 is committed 00002000:00010000:3.0:1789760760.243414:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22884352 -> 22888448 00010000:00010000:3.0:1789760760.243435:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22884352->22888447] (req 22884352->22888447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.243450:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22884352->22888447] (req 22884352->22888447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.243460:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22884352->22888447] (req 22884352->22888447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.243469:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760760.243475:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22884352->22888447] (req 22884352->22888447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.243487:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22884352->22888447] (req 22884352->22888447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.243515:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22884352->22888447] (req 22884352->22888447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.243524:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209282 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22884352->22888447] (req 22884352->22888447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760760.243687:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27226us (27618us total) trans 4294972905 rc 0/0 00000100:00100000:3.0:1789760760.243699:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5616 00000100:00100000:3.0:1789760760.243711:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749507938267 time 749547370182 state 0x2 ref 0 00000100:00100000:0.0:1789760760.249966:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760760.250059:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484282624 00000100:00100000:1.0:1789760760.250089:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484282624 00000100:00100000:1.0:1789760760.250172:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749507938267 time 749547370182 state 0x0 ref 1 00000100:00100000:1.0:1789760760.250198:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749547630598 time 749547370182 state 0x1 ref 1 00000100:00100000:1.0:1789760760.250233:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749547630598 time 749581163893 state 0x0 ref 1 00000100:00100000:1.0:1789760760.250247:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760760.250313:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22888448, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:1.0:1789760760.250324:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.250343:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:1.0:1789760760.250348:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760760.250356:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22888448->22892543] (req 22888448->22892543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.250377:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22888448->22892543] (req 22888448->22892543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760760.281816:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972906 is committed 00002000:00010000:1.0:1789760760.282152:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22888448 -> 22892544 00010000:00010000:1.0:1789760760.282183:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22888448->22892543] (req 22888448->22892543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.282199:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22888448->22892543] (req 22888448->22892543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.282271:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22888448->22892543] (req 22888448->22892543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.282280:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760760.282286:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22888448->22892543] (req 22888448->22892543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.282296:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22888448->22892543] (req 22888448->22892543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.282326:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22888448->22892543] (req 22888448->22892543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.282335:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209289 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22888448->22892543] (req 22888448->22892543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760760.282508:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32256us (32557us total) trans 4294972906 rc 0/0 00000100:00100000:1.0:1789760760.282521:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5617 00000100:00100000:1.0:1789760760.282532:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749547630598 time 749581163893 state 0x2 ref 0 00000100:00100000:2.0:1789760760.293893:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760760.294621:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484282752 00000100:00100000:0.0:1789760760.294635:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484282752 00000100:00100000:0.0:1789760760.294793:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749547630598 time 749581163893 state 0x0 ref 1 00000100:00100000:0.0:1789760760.294813:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749581424309 time 749581163893 state 0x1 ref 1 00000100:00100000:0.0:1789760760.294965:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749581424309 time 749625886380 state 0x0 ref 1 00000100:00100000:0.0:1789760760.294983:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760760.295059:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22892544, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:0.0:1789760760.295116:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.295136:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:0.0:1789760760.295142:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760760.295150:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22892544->22896639] (req 22892544->22896639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.295185:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22892544->22896639] (req 22892544->22896639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760760.332469:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972907 is committed 00002000:00010000:3.0:1789760760.332657:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22892544 -> 22896640 00010000:00010000:3.0:1789760760.332678:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22892544->22896639] (req 22892544->22896639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.332693:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22892544->22896639] (req 22892544->22896639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.332703:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22892544->22896639] (req 22892544->22896639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.332712:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760760.332719:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22892544->22896639] (req 22892544->22896639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.332729:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22892544->22896639] (req 22892544->22896639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.332754:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22892544->22896639] (req 22892544->22896639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.332763:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209290 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22892544->22896639] (req 22892544->22896639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760760.332901:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37914us (39078us total) trans 4294972907 rc 0/0 00000100:00100000:3.0:1789760760.332914:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5618 00000100:00100000:3.0:1789760760.332928:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749581424309 time 749625886380 state 0x2 ref 0 00000100:00100000:2.0:1789760760.335737:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760760.335898:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484282880 00000100:00100000:0.0:1789760760.335912:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484282880 00000100:00100000:0.0:1789760760.336007:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749581424309 time 749625886380 state 0x0 ref 1 00000100:00100000:0.0:1789760760.336029:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749626146796 time 749625886380 state 0x1 ref 1 00000100:00100000:0.0:1789760760.336096:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749626146796 time 749666994314 state 0x0 ref 1 00000100:00100000:0.0:1789760760.336113:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760760.336196:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22896640, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:0.0:1789760760.336329:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.336353:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:0.0:1789760760.336359:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760760.336367:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22896640->22900735] (req 22896640->22900735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.336386:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22896640->22900735] (req 22896640->22900735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760760.370611:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972908 is committed 00002000:00010000:3.0:1789760760.370905:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22896640 -> 22900736 00010000:00010000:3.0:1789760760.370924:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22896640->22900735] (req 22896640->22900735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.370939:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22896640->22900735] (req 22896640->22900735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.370947:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22896640->22900735] (req 22896640->22900735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.370970:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760760.370975:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22896640->22900735] (req 22896640->22900735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.370985:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22896640->22900735] (req 22896640->22900735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.371010:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22896640->22900735] (req 22896640->22900735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.371019:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209297 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22896640->22900735] (req 22896640->22900735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760760.371185:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484282880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35061us (35449us total) trans 4294972908 rc 0/0 00000100:00100000:3.0:1789760760.371201:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5619 00000100:00100000:3.0:1789760760.371214:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749626146796 time 749666994314 state 0x2 ref 0 00000100:00100000:0.0:1789760760.375614:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760760.375798:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484283008 00000100:00100000:3.0:1789760760.375814:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484283008 00000100:00100000:3.0:1789760760.375903:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749626146796 time 749666994314 state 0x0 ref 1 00000100:00100000:3.0:1789760760.375923:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749667254730 time 749666994314 state 0x1 ref 1 00000100:00100000:3.0:1789760760.376153:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749667254730 time 749707081462 state 0x0 ref 1 00000100:00100000:3.0:1789760760.376184:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484283008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760760.376253:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22900736, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:3.0:1789760760.376264:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.376294:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:3.0:1789760760.376300:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760760.376307:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22900736->22904831] (req 22900736->22904831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.376325:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22900736->22904831] (req 22900736->22904831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760760.420365:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972909 is committed 00002000:00010000:1.0:1789760760.420449:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22900736 -> 22904832 00010000:00010000:1.0:1789760760.420470:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22900736->22904831] (req 22900736->22904831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.420485:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22900736->22904831] (req 22900736->22904831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.420496:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22900736->22904831] (req 22900736->22904831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.420504:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760760.420510:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22900736->22904831] (req 22900736->22904831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.420521:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22900736->22904831] (req 22900736->22904831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.420546:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22900736->22904831] (req 22900736->22904831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.420562:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620929e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22900736->22904831] (req 22900736->22904831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760760.420697:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484283008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44520us (45091us total) trans 4294972909 rc 0/0 00000100:00100000:1.0:1789760760.420710:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5620 00000100:00100000:1.0:1789760760.420723:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749667254730 time 749707081462 state 0x2 ref 0 00000100:00100000:2.0:1789760760.423858:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760760.424340:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484283136 00000100:00100000:1.0:1789760760.424353:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484283136 00000100:00100000:2.0:1789760760.424554:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749667254730 time 749707081462 state 0x0 ref 1 00000100:00100000:2.0:1789760760.424574:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749707341878 time 749707081462 state 0x1 ref 1 00000100:00100000:2.0:1789760760.424650:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749707341878 time 749755579559 state 0x0 ref 1 00000100:00100000:2.0:1789760760.424666:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484283136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760760.424732:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22904832, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:2.0:1789760760.424743:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.424762:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:2.0:1789760760.424769:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760760.424777:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22904832->22908927] (req 22904832->22908927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.424793:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22904832->22908927] (req 22904832->22908927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760760.460657:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22904832 -> 22908928 00010000:00010000:3.0:1789760760.460686:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22904832->22908927] (req 22904832->22908927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.460701:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22904832->22908927] (req 22904832->22908927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.460712:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22904832->22908927] (req 22904832->22908927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.460721:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760760.460726:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22904832->22908927] (req 22904832->22908927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.460738:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22904832->22908927] (req 22904832->22908927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.460767:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22904832->22908927] (req 22904832->22908927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.460774:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962092a5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22904832->22908927] (req 22904832->22908927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760760.460895:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484283136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36225us (37045us total) trans 4294972910 rc 0/0 00000100:00100000:3.0:1789760760.460907:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5621 00000100:00100000:3.0:1789760760.460921:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749707341878 time 749755579559 state 0x2 ref 0 00000001:00080000:0.0:1789760760.461640:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972910 is committed 00000100:00100000:0.0:1789760760.468001:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760760.468111:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484283264 00000100:00100000:3.0:1789760760.468124:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484283264 00000100:00100000:3.0:1789760760.468216:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749707341878 time 749755579559 state 0x0 ref 1 00000100:00100000:3.0:1789760760.468235:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749755839975 time 749755579559 state 0x1 ref 1 00000100:00100000:3.0:1789760760.468276:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749755839975 time 749799206313 state 0x0 ref 1 00000100:00100000:3.0:1789760760.468289:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484283264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760760.468360:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22908928, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:3.0:1789760760.468371:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.468392:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:3.0:1789760760.468397:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760760.468404:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22908928->22913023] (req 22908928->22913023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.468422:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22908928->22913023] (req 22908928->22913023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760760.527430:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972911 is committed 00002000:00010000:3.0:1789760760.527602:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22908928 -> 22913024 00010000:00010000:3.0:1789760760.527624:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22908928->22913023] (req 22908928->22913023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.527640:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22908928->22913023] (req 22908928->22913023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.527650:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22908928->22913023] (req 22908928->22913023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.527659:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760760.527665:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22908928->22913023] (req 22908928->22913023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.527676:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22908928->22913023] (req 22908928->22913023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.527702:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22908928->22913023] (req 22908928->22913023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.527713:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092ac lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22908928->22913023] (req 22908928->22913023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760760.527883:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484283264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59588us (59889us total) trans 4294972911 rc 0/0 00000100:00100000:3.0:1789760760.527897:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5622 00000100:00100000:3.0:1789760760.527920:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749755839975 time 749799206313 state 0x2 ref 0 00000100:00100000:1.0:1789760760.532640:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760760.532776:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484283392 00000100:00100000:3.0:1789760760.532789:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484283392 00000100:00100000:3.0:1789760760.532859:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749755839975 time 749799206313 state 0x0 ref 1 00000100:00100000:3.0:1789760760.532877:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749799466729 time 749799206313 state 0x1 ref 1 00000100:00100000:3.0:1789760760.532909:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749799466729 time 749863840064 state 0x0 ref 1 00000100:00100000:3.0:1789760760.532924:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484283392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760760.532989:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22913024, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:3.0:1789760760.533011:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.533032:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:3.0:1789760760.533038:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760760.533050:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22913024->22917119] (req 22913024->22917119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.533103:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22913024->22917119] (req 22913024->22917119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760760.580811:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972912 is committed 00002000:00010000:0.0:1789760760.580947:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22913024 -> 22917120 00010000:00010000:0.0:1789760760.581142:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22913024->22917119] (req 22913024->22917119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.581158:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22913024->22917119] (req 22913024->22917119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.581169:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22913024->22917119] (req 22913024->22917119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.581188:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.581194:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22913024->22917119] (req 22913024->22917119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.581204:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22913024->22917119] (req 22913024->22917119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.581230:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22913024->22917119] (req 22913024->22917119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.581239:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092b3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22913024->22917119] (req 22913024->22917119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.581567:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484283392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48637us (48932us total) trans 4294972912 rc 0/0 00000100:00100000:0.0:1789760760.581580:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5623 00000100:00100000:0.0:1789760760.581591:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749799466729 time 749863840064 state 0x2 ref 0 00000100:00100000:2.0:1789760760.585326:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760760.588834:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484283520 00000100:00100000:1.0:1789760760.588855:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484283520 00000100:00100000:1.0:1789760760.588952:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749799466729 time 749863840064 state 0x0 ref 1 00000100:00100000:1.0:1789760760.588982:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749864100480 time 749863840064 state 0x1 ref 1 00000100:00100000:1.0:1789760760.589023:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749864100480 time 749919953545 state 0x0 ref 1 00000100:00100000:1.0:1789760760.589040:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484283520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760760.589141:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22917120, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:1.0:1789760760.589152:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.589182:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:1.0:1789760760.589188:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760760.589196:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22917120->22921215] (req 22917120->22921215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.589213:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22917120->22921215] (req 22917120->22921215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760760.638005:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972913 is committed 00002000:00010000:0.0:1789760760.638344:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22917120 -> 22921216 00010000:00010000:0.0:1789760760.638366:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22917120->22921215] (req 22917120->22921215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.638381:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22917120->22921215] (req 22917120->22921215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.638391:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22917120->22921215] (req 22917120->22921215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.638410:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.638416:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22917120->22921215] (req 22917120->22921215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.638425:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22917120->22921215] (req 22917120->22921215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.638452:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22917120->22921215] (req 22917120->22921215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.638466:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962092ba lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22917120->22921215] (req 22917120->22921215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.638595:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484283520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49555us (53278us total) trans 4294972913 rc 0/0 00000100:00100000:0.0:1789760760.638606:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5624 00000100:00100000:0.0:1789760760.638616:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749864100480 time 749919953545 state 0x2 ref 0 00000100:00100000:2.0:1789760760.644832:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760760.645221:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484283648 00000100:00100000:3.0:1789760760.645239:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484283648 00000100:00100000:3.0:1789760760.645332:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749864100480 time 749919953545 state 0x0 ref 1 00000100:00100000:3.0:1789760760.645350:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749920213961 time 749919953545 state 0x1 ref 1 00000100:00100000:3.0:1789760760.645385:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749920213961 time 749976315980 state 0x0 ref 1 00000100:00100000:3.0:1789760760.645402:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abd180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484283648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760760.645482:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22921216, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:3.0:1789760760.645493:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.645514:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=749s v=(0 0) 00010000:00010000:3.0:1789760760.645519:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760760.645527:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22921216->22925311] (req 22921216->22925311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.645546:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22921216->22925311] (req 22921216->22925311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760760.691036:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972914 is committed 00002000:00010000:2.0:1789760760.691379:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22921216 -> 22925312 00010000:00010000:2.0:1789760760.691394:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22921216->22925311] (req 22921216->22925311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.691407:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22921216->22925311] (req 22921216->22925311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.691417:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22921216->22925311] (req 22921216->22925311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.691425:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760760.691430:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22921216->22925311] (req 22921216->22925311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.691440:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22921216->22925311] (req 22921216->22925311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.691462:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22921216->22925311] (req 22921216->22925311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.691470:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962092c1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22921216->22925311] (req 22921216->22925311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760760.691608:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abd180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484283648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46206us (46779us total) trans 4294972914 rc 0/0 00000100:00100000:2.0:1789760760.691620:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5625 00000100:00100000:2.0:1789760760.691630:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749920213961 time 749976315980 state 0x2 ref 0 00000100:00100000:2.0:1789760760.700618:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760760.700928:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484283776 00000100:00100000:0.0:1789760760.700940:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484283776 00000100:00100000:0.0:1789760760.701022:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749920213961 time 749976315980 state 0x0 ref 1 00000100:00100000:0.0:1789760760.701040:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749976576396 time 749976315980 state 0x1 ref 1 00000100:00100000:0.0:1789760760.701103:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749976576396 time 750032033156 state 0x0 ref 1 00000100:00100000:0.0:1789760760.701118:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484283776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760760.701181:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22925312, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:0.0:1789760760.701191:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.701208:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:0.0:1789760760.701212:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760760.701218:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22925312->22929407] (req 22925312->22929407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.701233:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22925312->22929407] (req 22925312->22929407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.716381:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760760.737759:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972915 is committed 00002000:00010000:0.0:1789760760.737841:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22925312 -> 22929408 00010000:00010000:0.0:1789760760.737856:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22925312->22929407] (req 22925312->22929407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.737869:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22925312->22929407] (req 22925312->22929407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.737878:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22925312->22929407] (req 22925312->22929407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.737885:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.737890:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22925312->22929407] (req 22925312->22929407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.737898:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22925312->22929407] (req 22925312->22929407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.737914:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22925312->22929407] (req 22925312->22929407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.737921:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092c8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22925312->22929407] (req 22925312->22929407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.738352:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484283776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37233us (37738us total) trans 4294972915 rc 0/0 00000100:00100000:0.0:1789760760.738362:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5626 00000100:00100000:0.0:1789760760.738372:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749976576396 time 750032033156 state 0x2 ref 0 00000100:00100000:3.0:1789760760.742637:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760760.745207:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484283904 00000100:00100000:0.0:1789760760.745228:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484283904 00000100:00100000:0.0:1789760760.746007:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 749976576396 time 750032033156 state 0x0 ref 1 00000100:00100000:0.0:1789760760.746029:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750032293572 time 750032033156 state 0x1 ref 1 00000100:00100000:0.0:1789760760.746105:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750032293572 time 750077034961 state 0x0 ref 1 00000100:00100000:0.0:1789760760.746124:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba77100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484283904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760760.746198:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22929408, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:0.0:1789760760.746209:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.746229:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:0.0:1789760760.746235:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760760.746243:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22929408->22933503] (req 22929408->22933503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.746271:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22929408->22933503] (req 22929408->22933503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760760.802554:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972916 is committed 00002000:00010000:0.0:1789760760.802664:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22929408 -> 22933504 00010000:00010000:0.0:1789760760.802685:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22929408->22933503] (req 22929408->22933503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.802701:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22929408->22933503] (req 22929408->22933503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.802712:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22929408->22933503] (req 22929408->22933503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.802721:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.802726:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22929408->22933503] (req 22929408->22933503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.802806:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22929408->22933503] (req 22929408->22933503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.802831:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22929408->22933503] (req 22929408->22933503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.802840:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962092cf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22929408->22933503] (req 22929408->22933503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.803248:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba77100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484283904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57122us (60618us total) trans 4294972916 rc 0/0 00000100:00100000:0.0:1789760760.803265:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5627 00000100:00100000:0.0:1789760760.803278:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750032293572 time 750077034961 state 0x2 ref 0 00000100:00100000:1.0:1789760760.806361:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760760.806457:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484284032 00000100:00100000:0.0:1789760760.806469:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484284032 00000100:00100000:0.0:1789760760.806551:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750032293572 time 750077034961 state 0x0 ref 1 00000100:00100000:0.0:1789760760.806572:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750077295377 time 750077034961 state 0x1 ref 1 00000100:00100000:0.0:1789760760.806612:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750077295377 time 750137542093 state 0x0 ref 1 00000100:00100000:0.0:1789760760.806627:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484284032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760760.806696:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22933504, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:0.0:1789760760.806706:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.806725:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:0.0:1789760760.806743:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760760.806751:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22933504->22937599] (req 22933504->22937599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.806770:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22933504->22937599] (req 22933504->22937599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760760.845338:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22933504 -> 22937600 00010000:00010000:0.0:1789760760.845369:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22933504->22937599] (req 22933504->22937599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.845387:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22933504->22937599] (req 22933504->22937599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.845397:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22933504->22937599] (req 22933504->22937599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.845406:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.845412:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22933504->22937599] (req 22933504->22937599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.845423:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22933504->22937599] (req 22933504->22937599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.845450:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22933504->22937599] (req 22933504->22937599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.845459:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092d6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22933504->22937599] (req 22933504->22937599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.845583:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484284032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38951us (39240us total) trans 4294972917 rc 0/0 00000100:00100000:0.0:1789760760.845596:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5628 00000100:00100000:0.0:1789760760.845608:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750077295377 time 750137542093 state 0x2 ref 0 00000001:00080000:2.0:1789760760.846481:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972917 is committed 00000100:00100000:2.0:1789760760.849294:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760760.849541:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484284160 00000100:00100000:3.0:1789760760.849556:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484284160 00000100:00100000:3.0:1789760760.849618:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750077295377 time 750137542093 state 0x0 ref 1 00000100:00100000:3.0:1789760760.849632:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750137802509 time 750137542093 state 0x1 ref 1 00000100:00100000:3.0:1789760760.849659:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750137802509 time 750180591223 state 0x0 ref 1 00000100:00100000:3.0:1789760760.849671:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484284160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760760.849732:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22937600, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:3.0:1789760760.849740:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.849757:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:3.0:1789760760.849762:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760760.849768:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22937600->22941695] (req 22937600->22941695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.849782:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22937600->22941695] (req 22937600->22941695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760760.897010:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22937600 -> 22941696 00010000:00010000:0.0:1789760760.897038:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22937600->22941695] (req 22937600->22941695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.897054:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22937600->22941695] (req 22937600->22941695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.897094:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22937600->22941695] (req 22937600->22941695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.897103:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.897108:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22937600->22941695] (req 22937600->22941695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.897118:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22937600->22941695] (req 22937600->22941695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.897142:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22937600->22941695] (req 22937600->22941695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.897149:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962092dd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22937600->22941695] (req 22937600->22941695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.897275:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484284160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47597us (47995us total) trans 4294972918 rc 0/0 00000100:00100000:0.0:1789760760.897288:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5629 00000100:00100000:0.0:1789760760.897302:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750137802509 time 750180591223 state 0x2 ref 0 00000001:00080000:1.0:1789760760.897455:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972918 is committed 00000100:00100000:2.0:1789760760.901551:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760760.901719:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484284288 00000100:00100000:3.0:1789760760.901732:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484284288 00000100:00100000:3.0:1789760760.901801:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750137802509 time 750180591223 state 0x0 ref 1 00000100:00100000:3.0:1789760760.901818:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750180851639 time 750180591223 state 0x1 ref 1 00000100:00100000:3.0:1789760760.901880:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750180851639 time 750232809886 state 0x0 ref 1 00000100:00100000:3.0:1789760760.901895:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484284288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760760.901958:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22941696, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:3.0:1789760760.901969:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.901990:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:3.0:1789760760.901995:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760760.902003:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22941696->22945791] (req 22941696->22945791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760760.902019:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22941696->22945791] (req 22941696->22945791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760760.929423:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972919 is committed 00002000:00010000:0.0:1789760760.929531:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22941696 -> 22945792 00010000:00010000:0.0:1789760760.929551:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22941696->22945791] (req 22941696->22945791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.929566:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22941696->22945791] (req 22941696->22945791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.929576:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22941696->22945791] (req 22941696->22945791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.929584:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760760.929589:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22941696->22945791] (req 22941696->22945791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.929600:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22941696->22945791] (req 22941696->22945791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.929623:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22941696->22945791] (req 22941696->22945791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.929630:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962092e4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22941696->22945791] (req 22941696->22945791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760760.929743:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484284288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27844us (28200us total) trans 4294972919 rc 0/0 00000100:00100000:0.0:1789760760.929755:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5630 00000100:00100000:0.0:1789760760.929768:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750180851639 time 750232809886 state 0x2 ref 0 00000100:00100000:2.0:1789760760.932954:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760760.933727:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484284416 00000100:00100000:0.0:1789760760.933742:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484284416 00000100:00100000:0.0:1789760760.933826:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750180851639 time 750232809886 state 0x0 ref 1 00000100:00100000:0.0:1789760760.933846:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750233070302 time 750232809886 state 0x1 ref 1 00000100:00100000:0.0:1789760760.933939:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750233070302 time 750264868707 state 0x0 ref 1 00000100:00100000:0.0:1789760760.934226:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484284416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760760.934296:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22945792, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:0.0:1789760760.934307:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.934327:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:0.0:1789760760.934332:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760760.934349:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22945792->22949887] (req 22945792->22949887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760760.934367:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22945792->22949887] (req 22945792->22949887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760760.969033:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22945792 -> 22949888 00010000:00010000:2.0:1789760760.969062:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22945792->22949887] (req 22945792->22949887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760760.969166:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972920 is committed 00010000:00010000:2.0:1789760760.969243:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22945792->22949887] (req 22945792->22949887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.969254:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22945792->22949887] (req 22945792->22949887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.969262:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760760.969269:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22945792->22949887] (req 22945792->22949887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.969279:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22945792->22949887] (req 22945792->22949887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.969306:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22945792->22949887] (req 22945792->22949887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760760.969315:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962092eb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22945792->22949887] (req 22945792->22949887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760760.969483:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484284416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35525us (36536us total) trans 4294972920 rc 0/0 00000100:00100000:2.0:1789760760.969496:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5631 00000100:00100000:2.0:1789760760.969508:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750233070302 time 750264868707 state 0x2 ref 0 00000100:00100000:2.0:1789760760.978981:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760760.979364:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484284544 00000100:00100000:0.0:1789760760.979379:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484284544 00000100:00100000:1.0:1789760760.979542:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750233070302 time 750264868707 state 0x0 ref 1 00000100:00100000:1.0:1789760760.979561:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750265129123 time 750264868707 state 0x1 ref 1 00000100:00100000:1.0:1789760760.979624:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750265129123 time 750310554306 state 0x0 ref 1 00000100:00100000:1.0:1789760760.979637:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484284544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760760.979699:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22949888, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:1.0:1789760760.979710:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.979729:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:1.0:1789760760.979733:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760760.979740:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22949888->22953983] (req 22949888->22953983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760760.979762:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22949888->22953983] (req 22949888->22953983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760761.016381:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972921 is committed 00002000:00010000:0.0:1789760761.016499:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22949888 -> 22953984 00010000:00010000:0.0:1789760761.016520:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22949888->22953983] (req 22949888->22953983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.016542:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22949888->22953983] (req 22949888->22953983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.016553:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22949888->22953983] (req 22949888->22953983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.016560:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760761.016566:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22949888->22953983] (req 22949888->22953983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.016577:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22949888->22953983] (req 22949888->22953983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.016603:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22949888->22953983] (req 22949888->22953983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.016611:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962092f2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22949888->22953983] (req 22949888->22953983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760761.016785:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484284544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37143us (37828us total) trans 4294972921 rc 0/0 00000100:00100000:0.0:1789760761.016798:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5632 00000100:00100000:0.0:1789760761.016812:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750265129123 time 750310554306 state 0x2 ref 0 00000100:00100000:3.0:1789760761.019808:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760761.019913:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484284672 00000100:00100000:2.0:1789760761.019928:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484284672 00000100:00100000:2.0:1789760761.021265:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750265129123 time 750310554306 state 0x0 ref 1 00000100:00100000:2.0:1789760761.021355:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750310814722 time 750310554306 state 0x1 ref 1 00000100:00100000:2.0:1789760761.021581:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750310814722 time 750352510463 state 0x0 ref 1 00000100:00100000:2.0:1789760761.021606:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484284672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760761.021683:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22953984, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:2.0:1789760761.021694:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.021714:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:2.0:1789760761.021720:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760761.021727:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22953984->22958079] (req 22953984->22958079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.021745:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22953984->22958079] (req 22953984->22958079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760761.052231:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22953984 -> 22958080 00010000:00010000:2.0:1789760761.052257:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22953984->22958079] (req 22953984->22958079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.052338:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22953984->22958079] (req 22953984->22958079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.052350:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22953984->22958079] (req 22953984->22958079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.052358:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760761.052364:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22953984->22958079] (req 22953984->22958079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.052375:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22953984->22958079] (req 22953984->22958079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.052402:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22953984->22958079] (req 22953984->22958079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.052411:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962092f9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22953984->22958079] (req 22953984->22958079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760761.052531:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484284672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30930us (32730us total) trans 4294972922 rc 0/0 00000100:00100000:2.0:1789760761.052543:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5633 00000100:00100000:2.0:1789760761.052556:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750310814722 time 750352510463 state 0x2 ref 0 00000001:00080000:0.0:1789760761.052641:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972922 is committed 00000100:00100000:2.0:1789760761.055732:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760761.055840:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484284800 00000100:00100000:0.0:1789760761.055853:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484284800 00000100:00100000:0.0:1789760761.055937:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750310814722 time 750352510463 state 0x0 ref 1 00000100:00100000:0.0:1789760761.055958:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750352770879 time 750352510463 state 0x1 ref 1 00000100:00100000:0.0:1789760761.055999:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750352770879 time 750386929669 state 0x0 ref 1 00000100:00100000:0.0:1789760761.056015:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484284800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760761.056120:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22958080, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:0.0:1789760761.056131:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.056149:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:0.0:1789760761.056155:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760761.056162:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22958080->22962175] (req 22958080->22962175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.056194:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22958080->22962175] (req 22958080->22962175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760761.148711:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972923 is committed 00002000:00010000:3.0:1789760761.149751:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22958080 -> 22962176 00010000:00010000:3.0:1789760761.149786:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22958080->22962175] (req 22958080->22962175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.149801:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22958080->22962175] (req 22958080->22962175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.149812:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22958080->22962175] (req 22958080->22962175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.149826:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760761.149832:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22958080->22962175] (req 22958080->22962175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.149842:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22958080->22962175] (req 22958080->22962175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.149865:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22958080->22962175] (req 22958080->22962175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.149875:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209300 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22958080->22962175] (req 22958080->22962175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760761.150009:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484284800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 93992us (94286us total) trans 4294972923 rc 0/0 00000100:00100000:3.0:1789760761.150021:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5634 00000100:00100000:3.0:1789760761.150033:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750352770879 time 750386929669 state 0x2 ref 0 00000100:00100000:3.0:1789760761.152765:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760761.152867:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484284928 00000100:00100000:0.0:1789760761.152882:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484284928 00000100:00100000:0.0:1789760761.152974:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750352770879 time 750386929669 state 0x0 ref 1 00000100:00100000:0.0:1789760761.152993:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750387190085 time 750386929669 state 0x1 ref 1 00000100:00100000:0.0:1789760761.153032:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750387190085 time 750483962679 state 0x0 ref 1 00000100:00100000:0.0:1789760761.153047:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484284928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760761.153148:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22962176, mtime: 0x6aad94f8, atime: 0x0, ctime: 0x6aad94f8, blocks: 0xb007 00010000:00010000:0.0:1789760761.153159:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.153189:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:0.0:1789760761.153194:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760761.153202:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22962176->22966271] (req 22962176->22966271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.153219:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22962176->22966271] (req 22962176->22966271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760761.195463:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22962176 -> 22966272 00002000:00010000:3.0:1789760761.195474:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760760 -> 1789760761 00002000:00010000:3.0:1789760761.195477:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760760 -> 1789760761 00010000:00010000:3.0:1789760761.195494:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22962176->22966271] (req 22962176->22966271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.195516:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22962176->22966271] (req 22962176->22966271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.195525:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22962176->22966271] (req 22962176->22966271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.195533:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760761.195538:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22962176->22966271] (req 22962176->22966271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.195551:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22962176->22966271] (req 22962176->22966271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.195574:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22962176->22966271] (req 22962176->22966271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.195582:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209307 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22962176->22966271] (req 22962176->22966271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760761.195712:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484284928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42658us (42950us total) trans 4294972924 rc 0/0 00000100:00100000:3.0:1789760761.195727:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5635 00000100:00100000:3.0:1789760761.195740:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750387190085 time 750483962679 state 0x2 ref 0 00000001:00080000:0.0:1789760761.196494:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972924 is committed 00000100:00100000:2.0:1789760761.199698:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760761.199816:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484285056 00000100:00100000:1.0:1789760761.199829:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484285056 00000100:00100000:1.0:1789760761.199904:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750387190085 time 750483962679 state 0x0 ref 1 00000100:00100000:1.0:1789760761.199924:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750484223095 time 750483962679 state 0x1 ref 1 00000100:00100000:1.0:1789760761.199987:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750484223095 time 750530917781 state 0x0 ref 1 00000100:00100000:1.0:1789760761.200004:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484285056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760761.200106:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22966272, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:1.0:1789760761.200118:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.200137:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:1.0:1789760761.200142:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760761.200150:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22966272->22970367] (req 22966272->22970367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.200166:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22966272->22970367] (req 22966272->22970367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760761.266402:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22966272 -> 22970368 00010000:00010000:0.0:1789760761.266441:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22966272->22970367] (req 22966272->22970367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.266457:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22966272->22970367] (req 22966272->22970367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.266467:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22966272->22970367] (req 22966272->22970367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.266481:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760761.266487:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22966272->22970367] (req 22966272->22970367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.266497:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22966272->22970367] (req 22966272->22970367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.266521:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22966272->22970367] (req 22966272->22970367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.266528:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620930e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22966272->22970367] (req 22966272->22970367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760761.266649:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484285056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66642us (66956us total) trans 4294972925 rc 0/0 00000100:00100000:0.0:1789760761.266662:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5636 00000100:00100000:0.0:1789760761.266675:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750484223095 time 750530917781 state 0x2 ref 0 00000001:00080000:3.0:1789760761.267769:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972925 is committed 00000100:00100000:2.0:1789760761.270516:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760761.270897:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484285184 00000100:00100000:0.0:1789760761.270910:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484285184 00000100:00100000:0.0:1789760761.271002:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750484223095 time 750530917781 state 0x0 ref 1 00000100:00100000:0.0:1789760761.271021:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750531178197 time 750530917781 state 0x1 ref 1 00000100:00100000:0.0:1789760761.271051:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750531178197 time 750601982154 state 0x0 ref 1 00000100:00100000:0.0:1789760761.271111:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484285184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760761.271180:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22970368, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:0.0:1789760761.271190:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.271219:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:0.0:1789760761.271224:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760761.271231:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22970368->22974463] (req 22970368->22974463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.271249:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22970368->22974463] (req 22970368->22974463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760761.316916:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972926 is committed 00002000:00010000:3.0:1789760761.317418:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22970368 -> 22974464 00010000:00010000:3.0:1789760761.317439:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22970368->22974463] (req 22970368->22974463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.317458:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22970368->22974463] (req 22970368->22974463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.317468:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22970368->22974463] (req 22970368->22974463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.317476:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760761.317482:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22970368->22974463] (req 22970368->22974463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.317492:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22970368->22974463] (req 22970368->22974463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.317519:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22970368->22974463] (req 22970368->22974463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.317528:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209315 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22970368->22974463] (req 22970368->22974463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760761.317654:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484285184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46580us (47150us total) trans 4294972926 rc 0/0 00000100:00100000:3.0:1789760761.317668:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5637 00000100:00100000:3.0:1789760761.317679:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750531178197 time 750601982154 state 0x2 ref 0 00000100:00100000:2.0:1789760761.320322:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760761.320429:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484285312 00000100:00100000:3.0:1789760761.320444:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484285312 00000100:00100000:3.0:1789760761.320519:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750531178197 time 750601982154 state 0x0 ref 1 00000100:00100000:3.0:1789760761.320538:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750602242570 time 750601982154 state 0x1 ref 1 00000100:00100000:3.0:1789760761.320611:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750602242570 time 750651541029 state 0x0 ref 1 00000100:00100000:3.0:1789760761.320625:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484285312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760761.320686:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22974464, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:3.0:1789760761.320697:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.320718:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:3.0:1789760761.320723:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760761.320730:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22974464->22978559] (req 22974464->22978559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.320746:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22974464->22978559] (req 22974464->22978559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760761.351790:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972927 is committed 00002000:00010000:3.0:1789760761.352230:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22974464 -> 22978560 00010000:00010000:3.0:1789760761.352251:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22974464->22978559] (req 22974464->22978559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.352265:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22974464->22978559] (req 22974464->22978559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.352274:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22974464->22978559] (req 22974464->22978559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.352282:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760761.352288:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22974464->22978559] (req 22974464->22978559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.352298:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22974464->22978559] (req 22974464->22978559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.352325:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22974464->22978559] (req 22974464->22978559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.352333:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620931c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22974464->22978559] (req 22974464->22978559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760761.352499:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484285312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31867us (32181us total) trans 4294972927 rc 0/0 00000100:00100000:3.0:1789760761.352514:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5638 00000100:00100000:3.0:1789760761.352526:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750602242570 time 750651541029 state 0x2 ref 0 00000100:00100000:3.0:1789760761.356899:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760761.357014:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484285440 00000100:00100000:2.0:1789760761.357028:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484285440 00000100:00100000:2.0:1789760761.357157:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750602242570 time 750651541029 state 0x0 ref 1 00000100:00100000:2.0:1789760761.357189:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750651801445 time 750651541029 state 0x1 ref 1 00000100:00100000:2.0:1789760761.357219:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750651801445 time 750688149445 state 0x0 ref 1 00000100:00100000:2.0:1789760761.357233:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484285440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760761.357306:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22978560, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:2.0:1789760761.357318:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.357478:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:2.0:1789760761.357485:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760761.357492:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22978560->22982655] (req 22978560->22982655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.357510:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22978560->22982655] (req 22978560->22982655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760761.400411:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972928 is committed 00002000:00010000:2.0:1789760761.400534:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22978560 -> 22982656 00010000:00010000:2.0:1789760761.400554:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22978560->22982655] (req 22978560->22982655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.400570:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22978560->22982655] (req 22978560->22982655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.400580:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22978560->22982655] (req 22978560->22982655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.400589:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760761.400595:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22978560->22982655] (req 22978560->22982655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.400606:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22978560->22982655] (req 22978560->22982655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.400635:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22978560->22982655] (req 22978560->22982655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.400644:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209323 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22978560->22982655] (req 22978560->22982655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760761.400769:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484285440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43532us (43878us total) trans 4294972928 rc 0/0 00000100:00100000:2.0:1789760761.400781:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5639 00000100:00100000:2.0:1789760761.400793:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750651801445 time 750688149445 state 0x2 ref 0 00000100:00100000:2.0:1789760761.406992:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760761.407328:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484285568 00000100:00100000:0.0:1789760761.407343:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484285568 00000100:00100000:0.0:1789760761.407432:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750651801445 time 750688149445 state 0x0 ref 1 00000100:00100000:0.0:1789760761.407452:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750688409861 time 750688149445 state 0x1 ref 1 00000100:00100000:0.0:1789760761.407548:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750688409861 time 750738477666 state 0x0 ref 1 00000100:00100000:0.0:1789760761.407567:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484285568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760761.407641:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22982656, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:0.0:1789760761.407653:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.407679:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:0.0:1789760761.407684:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760761.407691:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22982656->22986751] (req 22982656->22986751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.407710:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22982656->22986751] (req 22982656->22986751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760761.454907:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972929 is committed 00002000:00010000:1.0:1789760761.454941:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22982656 -> 22986752 00010000:00010000:1.0:1789760761.454960:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22982656->22986751] (req 22982656->22986751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.454975:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22982656->22986751] (req 22982656->22986751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.454985:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22982656->22986751] (req 22982656->22986751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.454992:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760761.454997:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22982656->22986751] (req 22982656->22986751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.455008:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22982656->22986751] (req 22982656->22986751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.455033:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22982656->22986751] (req 22982656->22986751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.455041:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620932a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22982656->22986751] (req 22982656->22986751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760761.455208:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484285568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47641us (48234us total) trans 4294972929 rc 0/0 00000100:00100000:1.0:1789760761.455221:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5640 00000100:00100000:1.0:1789760761.455233:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750688409861 time 750738477666 state 0x2 ref 0 00000100:00100000:3.0:1789760761.464535:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760761.464693:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484285696 00000100:00100000:0.0:1789760761.464707:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484285696 00000100:00100000:0.0:1789760761.464795:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750688409861 time 750738477666 state 0x0 ref 1 00000100:00100000:0.0:1789760761.464821:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750738738082 time 750738477666 state 0x1 ref 1 00000100:00100000:0.0:1789760761.464906:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750738738082 time 750795836207 state 0x0 ref 1 00000100:00100000:0.0:1789760761.464923:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484285696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760761.464990:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22986752, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:0.0:1789760761.465001:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.465022:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:0.0:1789760761.465027:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760761.465035:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22986752->22990847] (req 22986752->22990847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.465059:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22986752->22990847] (req 22986752->22990847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760761.508111:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22986752 -> 22990848 00010000:00010000:0.0:1789760761.508147:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22986752->22990847] (req 22986752->22990847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.508175:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22986752->22990847] (req 22986752->22990847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.508187:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22986752->22990847] (req 22986752->22990847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.508196:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760761.508202:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22986752->22990847] (req 22986752->22990847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.508212:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22986752->22990847] (req 22986752->22990847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.508237:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22986752->22990847] (req 22986752->22990847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.508245:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209331 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22986752->22990847] (req 22986752->22990847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760761.508435:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484285696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43508us (44145us total) trans 4294972930 rc 0/0 00000100:00100000:0.0:1789760761.508449:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5641 00000100:00100000:0.0:1789760761.508462:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750738738082 time 750795836207 state 0x2 ref 0 00000001:00080000:3.0:1789760761.508813:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972930 is committed 00000100:00100000:2.0:1789760761.512210:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760761.512474:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484285824 00000100:00100000:3.0:1789760761.512488:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484285824 00000100:00100000:3.0:1789760761.512568:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750738738082 time 750795836207 state 0x0 ref 1 00000100:00100000:3.0:1789760761.512589:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750796096623 time 750795836207 state 0x1 ref 1 00000100:00100000:3.0:1789760761.512625:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750796096623 time 750843555322 state 0x0 ref 1 00000100:00100000:3.0:1789760761.512639:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484285824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760761.512704:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22990848, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:3.0:1789760761.512715:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.512735:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:3.0:1789760761.512740:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760761.512748:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22990848->22994943] (req 22990848->22994943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.512765:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22990848->22994943] (req 22990848->22994943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760761.563880:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972931 is committed 00002000:00010000:1.0:1789760761.564160:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22990848 -> 22994944 00010000:00010000:1.0:1789760761.564191:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22990848->22994943] (req 22990848->22994943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.564206:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22990848->22994943] (req 22990848->22994943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.564215:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22990848->22994943] (req 22990848->22994943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.564223:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760761.564229:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22990848->22994943] (req 22990848->22994943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.564238:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22990848->22994943] (req 22990848->22994943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.564378:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22990848->22994943] (req 22990848->22994943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.564389:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209338 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22990848->22994943] (req 22990848->22994943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760761.564555:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484285824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51911us (52352us total) trans 4294972931 rc 0/0 00000100:00100000:1.0:1789760761.564572:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5642 00000100:00100000:1.0:1789760761.564585:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750796096623 time 750843555322 state 0x2 ref 0 00000100:00100000:3.0:1789760761.567834:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760761.568433:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484285952 00000100:00100000:2.0:1789760761.568448:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484285952 00000100:00100000:2.0:1789760761.568527:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750796096623 time 750843555322 state 0x0 ref 1 00000100:00100000:2.0:1789760761.568546:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750843815738 time 750843555322 state 0x1 ref 1 00000100:00100000:2.0:1789760761.568590:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750843815738 time 750899520783 state 0x0 ref 1 00000100:00100000:2.0:1789760761.568605:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98df100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484285952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760761.568671:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22994944, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:2.0:1789760761.568687:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.568714:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:2.0:1789760761.568719:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760761.568727:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22994944->22999039] (req 22994944->22999039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.568744:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22994944->22999039] (req 22994944->22999039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760761.626464:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972932 is committed 00002000:00010000:1.0:1789760761.626584:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22994944 -> 22999040 00010000:00010000:1.0:1789760761.626599:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22994944->22999039] (req 22994944->22999039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.626611:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22994944->22999039] (req 22994944->22999039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.626620:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22994944->22999039] (req 22994944->22999039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.626639:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760761.626645:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22994944->22999039] (req 22994944->22999039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.626655:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22994944->22999039] (req 22994944->22999039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.626676:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22994944->22999039] (req 22994944->22999039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.626684:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620933f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22994944->22999039] (req 22994944->22999039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760761.626783:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98df100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484285952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58177us (58960us total) trans 4294972932 rc 0/0 00000100:00100000:1.0:1789760761.626794:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5643 00000100:00100000:1.0:1789760761.626803:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750843815738 time 750899520783 state 0x2 ref 0 00000100:00100000:3.0:1789760761.637838:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760761.639649:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484286080 00000100:00100000:1.0:1789760761.639665:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484286080 00000100:00100000:1.0:1789760761.639732:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750843815738 time 750899520783 state 0x0 ref 1 00000100:00100000:1.0:1789760761.639747:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750899781199 time 750899520783 state 0x1 ref 1 00000100:00100000:1.0:1789760761.639784:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750899781199 time 750970714737 state 0x0 ref 1 00000100:00100000:1.0:1789760761.639797:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484286080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760761.639846:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 22999040, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:1.0:1789760761.639855:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.639871:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=750s v=(0 0) 00010000:00010000:1.0:1789760761.639876:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760761.639882:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22999040->23003135] (req 22999040->23003135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.639896:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22999040->23003135] (req 22999040->23003135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760761.702266:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 22999040 -> 23003136 00010000:00010000:2.0:1789760761.702286:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22999040->23003135] (req 22999040->23003135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.702298:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22999040->23003135] (req 22999040->23003135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.702308:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22999040->23003135] (req 22999040->23003135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.702315:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760761.702320:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22999040->23003135] (req 22999040->23003135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.702330:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22999040->23003135] (req 22999040->23003135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.702353:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22999040->23003135] (req 22999040->23003135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.702362:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209346 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [22999040->23003135] (req 22999040->23003135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760761.702459:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484286080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62659us (64741us total) trans 4294972933 rc 0/0 00000100:00100000:2.0:1789760761.702470:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5644 00000100:00100000:2.0:1789760761.702480:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750899781199 time 750970714737 state 0x2 ref 0 00000001:00080000:0.0:1789760761.703189:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972933 is committed 00000100:00100000:0.0:1789760761.709255:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760761.709340:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484286208 00000100:00100000:1.0:1789760761.709349:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484286208 00000100:00100000:1.0:1789760761.709405:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750899781199 time 750970714737 state 0x0 ref 1 00000100:00100000:1.0:1789760761.709420:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750970975153 time 750970714737 state 0x1 ref 1 00000100:00100000:1.0:1789760761.709452:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750970975153 time 751040382462 state 0x0 ref 1 00000100:00100000:1.0:1789760761.709463:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484286208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760761.709576:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23003136, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:1.0:1789760761.709585:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.709599:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:1.0:1789760761.709603:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760761.709610:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23003136->23007231] (req 23003136->23007231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.710548:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23003136->23007231] (req 23003136->23007231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.740323:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760761.815534:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972934 is committed 00002000:00010000:1.0:1789760761.815536:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23003136 -> 23007232 00010000:00010000:1.0:1789760761.815556:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23003136->23007231] (req 23003136->23007231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.815573:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23003136->23007231] (req 23003136->23007231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.815581:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23003136->23007231] (req 23003136->23007231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.815588:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760761.815592:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23003136->23007231] (req 23003136->23007231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.815600:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23003136->23007231] (req 23003136->23007231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.815618:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23003136->23007231] (req 23003136->23007231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.815626:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620934d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23003136->23007231] (req 23003136->23007231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760761.815719:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484286208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 106253us (106470us total) trans 4294972934 rc 0/0 00000100:00100000:1.0:1789760761.815729:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5645 00000100:00100000:1.0:1789760761.815739:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750970975153 time 751040382462 state 0x2 ref 0 00000100:00100000:1.0:1789760761.820843:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760761.820932:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484286336 00000100:00100000:3.0:1789760761.820943:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484286336 00000100:00100000:3.0:1789760761.821010:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 750970975153 time 751040382462 state 0x0 ref 1 00000100:00100000:3.0:1789760761.821028:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751040642878 time 751040382462 state 0x1 ref 1 00000100:00100000:3.0:1789760761.821060:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751040642878 time 751151990705 state 0x0 ref 1 00000100:00100000:3.0:1789760761.821115:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484286336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760761.821171:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23007232, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:3.0:1789760761.821181:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.821196:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:3.0:1789760761.821201:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760761.821206:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23007232->23011327] (req 23007232->23011327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760761.821219:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23007232->23011327] (req 23007232->23011327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760761.873132:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972935 is committed 00002000:00010000:0.0:1789760761.873256:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23007232 -> 23011328 00010000:00010000:0.0:1789760761.873273:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23007232->23011327] (req 23007232->23011327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.873286:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23007232->23011327] (req 23007232->23011327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.873295:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23007232->23011327] (req 23007232->23011327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.873303:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760761.873308:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23007232->23011327] (req 23007232->23011327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.873317:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23007232->23011327] (req 23007232->23011327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.873339:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23007232->23011327] (req 23007232->23011327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760761.873346:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209354 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23007232->23011327] (req 23007232->23011327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760761.873442:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484286336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52325us (52605us total) trans 4294972935 rc 0/0 00000100:00100000:0.0:1789760761.873452:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5646 00000100:00100000:0.0:1789760761.873462:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751040642878 time 751151990705 state 0x2 ref 0 00000100:00100000:2.0:1789760761.877037:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760761.877374:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484286464 00000100:00100000:1.0:1789760761.877385:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484286464 00000100:00100000:1.0:1789760761.877440:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751040642878 time 751151990705 state 0x0 ref 1 00000100:00100000:1.0:1789760761.877455:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751152251121 time 751151990705 state 0x1 ref 1 00000100:00100000:1.0:1789760761.877489:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751152251121 time 751208419699 state 0x0 ref 1 00000100:00100000:1.0:1789760761.877501:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484286464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760761.877773:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23011328, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:1.0:1789760761.877783:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.877799:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:1.0:1789760761.877803:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760761.877809:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23011328->23015423] (req 23011328->23015423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.877823:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23011328->23015423] (req 23011328->23015423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760761.930646:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972936 is committed 00002000:00010000:1.0:1789760761.930762:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23011328 -> 23015424 00010000:00010000:1.0:1789760761.930778:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23011328->23015423] (req 23011328->23015423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.931202:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23011328->23015423] (req 23011328->23015423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.931216:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23011328->23015423] (req 23011328->23015423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.931224:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760761.931229:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23011328->23015423] (req 23011328->23015423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.931238:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23011328->23015423] (req 23011328->23015423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.931323:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23011328->23015423] (req 23011328->23015423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.931332:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620935b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23011328->23015423] (req 23011328->23015423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760761.931468:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484286464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53964us (54435us total) trans 4294972936 rc 0/0 00000100:00100000:1.0:1789760761.931480:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5647 00000100:00100000:1.0:1789760761.931489:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751152251121 time 751208419699 state 0x2 ref 0 00000100:00100000:2.0:1789760761.935011:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760761.935216:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484286592 00000100:00100000:1.0:1789760761.935226:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484286592 00000100:00100000:1.0:1789760761.935343:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751152251121 time 751208419699 state 0x0 ref 1 00000100:00100000:1.0:1789760761.935358:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751208680115 time 751208419699 state 0x1 ref 1 00000100:00100000:1.0:1789760761.935430:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751208680115 time 751266360405 state 0x0 ref 1 00000100:00100000:1.0:1789760761.935443:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484286592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760761.935488:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23015424, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:1.0:1789760761.935497:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.935511:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:1.0:1789760761.935515:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760761.935520:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23015424->23019519] (req 23015424->23019519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.935533:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23015424->23019519] (req 23015424->23019519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760761.971783:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972937 is committed 00002000:00010000:1.0:1789760761.972475:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23015424 -> 23019520 00010000:00010000:1.0:1789760761.972491:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23015424->23019519] (req 23015424->23019519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.972505:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23015424->23019519] (req 23015424->23019519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.972515:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23015424->23019519] (req 23015424->23019519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.972522:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760761.972527:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23015424->23019519] (req 23015424->23019519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.972536:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23015424->23019519] (req 23015424->23019519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.972553:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23015424->23019519] (req 23015424->23019519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760761.972560:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209362 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23015424->23019519] (req 23015424->23019519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760761.972750:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484286592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37305us (37795us total) trans 4294972937 rc 0/0 00000100:00100000:1.0:1789760761.972763:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5648 00000100:00100000:1.0:1789760761.972773:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751208680115 time 751266360405 state 0x2 ref 0 00000100:00100000:0.0:1789760761.976414:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760761.976509:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484286720 00000100:00100000:2.0:1789760761.976518:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484286720 00000100:00100000:2.0:1789760761.976626:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751208680115 time 751266360405 state 0x0 ref 1 00000100:00100000:2.0:1789760761.976642:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751266620821 time 751266360405 state 0x1 ref 1 00000100:00100000:2.0:1789760761.976701:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751266620821 time 751307630443 state 0x0 ref 1 00000100:00100000:2.0:1789760761.976713:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484286720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760761.976759:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23019520, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:2.0:1789760761.976769:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.976791:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:2.0:1789760761.976795:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760761.976801:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23019520->23023615] (req 23019520->23023615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760761.976814:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23019520->23023615] (req 23019520->23023615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760762.016873:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972938 is committed 00002000:00010000:2.0:1789760762.017039:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23019520 -> 23023616 00010000:00010000:2.0:1789760762.017054:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23019520->23023615] (req 23019520->23023615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.017142:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23019520->23023615] (req 23019520->23023615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.017153:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23019520->23023615] (req 23019520->23023615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.017170:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760762.017176:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23019520->23023615] (req 23019520->23023615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.017185:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23019520->23023615] (req 23019520->23023615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.017203:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23019520->23023615] (req 23019520->23023615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.017210:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209369 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23019520->23023615] (req 23019520->23023615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760762.017473:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484286720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40755us (41060us total) trans 4294972938 rc 0/0 00000100:00100000:3.0:1789760762.017484:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5649 00000100:00100000:3.0:1789760762.017493:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751266620821 time 751307630443 state 0x2 ref 0 00000100:00100000:0.0:1789760762.026489:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760762.026583:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484286848 00000100:00100000:3.0:1789760762.026592:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484286848 00000100:00100000:3.0:1789760762.026834:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751266620821 time 751307630443 state 0x0 ref 1 00000100:00100000:3.0:1789760762.026854:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751307890859 time 751307630443 state 0x1 ref 1 00000100:00100000:3.0:1789760762.026888:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751307890859 time 751357818906 state 0x0 ref 1 00000100:00100000:3.0:1789760762.026901:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484286848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760762.027017:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23023616, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:3.0:1789760762.027027:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.027043:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:3.0:1789760762.027048:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760762.027054:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23023616->23027711] (req 23023616->23027711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.027089:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23023616->23027711] (req 23023616->23027711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760762.032002:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760762.095314:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972939 is committed 00002000:00010000:0.0:1789760762.095473:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23023616 -> 23027712 00010000:00010000:0.0:1789760762.095488:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23023616->23027711] (req 23023616->23027711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.095502:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23023616->23027711] (req 23023616->23027711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.095511:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23023616->23027711] (req 23023616->23027711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.095519:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760762.095524:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23023616->23027711] (req 23023616->23027711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.095533:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23023616->23027711] (req 23023616->23027711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.095556:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23023616->23027711] (req 23023616->23027711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.095563:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209370 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23023616->23027711] (req 23023616->23027711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760762.095662:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484286848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68759us (69179us total) trans 4294972939 rc 0/0 00000100:00100000:0.0:1789760762.095673:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5650 00000100:00100000:0.0:1789760762.095683:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751307890859 time 751357818906 state 0x2 ref 0 00000100:00100000:2.0:1789760762.105347:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760762.106126:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484286976 00000100:00100000:1.0:1789760762.106138:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484286976 00000100:00100000:1.0:1789760762.106207:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751307890859 time 751357818906 state 0x0 ref 1 00000100:00100000:1.0:1789760762.106223:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751358079322 time 751357818906 state 0x1 ref 1 00000100:00100000:1.0:1789760762.106258:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751358079322 time 751437188334 state 0x0 ref 1 00000100:00100000:1.0:1789760762.106270:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484286976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760762.106319:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23027712, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:1.0:1789760762.106328:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.106345:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:1.0:1789760762.106350:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760762.106357:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23027712->23031807] (req 23027712->23031807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.106372:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23027712->23031807] (req 23027712->23031807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760762.143037:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972940 is committed 00002000:00010000:3.0:1789760762.143215:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23027712 -> 23031808 00010000:00010000:3.0:1789760762.143229:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23027712->23031807] (req 23027712->23031807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.143241:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23027712->23031807] (req 23027712->23031807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.143250:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23027712->23031807] (req 23027712->23031807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.143262:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760762.143267:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23027712->23031807] (req 23027712->23031807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.143276:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23027712->23031807] (req 23027712->23031807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.143294:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23027712->23031807] (req 23027712->23031807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.143302:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209377 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23027712->23031807] (req 23027712->23031807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760762.143405:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484286976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37131us (38061us total) trans 4294972940 rc 0/0 00000100:00100000:3.0:1789760762.143417:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5651 00000100:00100000:3.0:1789760762.143428:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751358079322 time 751437188334 state 0x2 ref 0 00000100:00100000:2.0:1789760762.151477:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760762.151581:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484287104 00000100:00100000:1.0:1789760762.151591:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484287104 00000100:00100000:1.0:1789760762.151648:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751358079322 time 751437188334 state 0x0 ref 1 00000100:00100000:1.0:1789760762.151664:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751437448750 time 751437188334 state 0x1 ref 1 00000100:00100000:1.0:1789760762.151754:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751437448750 time 751482683985 state 0x0 ref 1 00000100:00100000:1.0:1789760762.151769:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484287104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760762.151821:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23031808, mtime: 0x6aad94f9, atime: 0x0, ctime: 0x6aad94f9, blocks: 0xb007 00010000:00010000:1.0:1789760762.151830:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.151849:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:1.0:1789760762.151854:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760762.151861:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23031808->23035903] (req 23031808->23035903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.151877:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23031808->23035903] (req 23031808->23035903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760762.188310:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972941 is committed 00002000:00010000:1.0:1789760762.188429:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23031808 -> 23035904 00002000:00010000:1.0:1789760762.188433:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760761 -> 1789760762 00002000:00010000:1.0:1789760762.188435:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760761 -> 1789760762 00010000:00010000:1.0:1789760762.188448:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23031808->23035903] (req 23031808->23035903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.188459:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23031808->23035903] (req 23031808->23035903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.188468:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23031808->23035903] (req 23031808->23035903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.188475:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760762.188480:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23031808->23035903] (req 23031808->23035903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.188490:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23031808->23035903] (req 23031808->23035903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.188509:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23031808->23035903] (req 23031808->23035903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.188522:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620937e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23031808->23035903] (req 23031808->23035903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760762.188617:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484287104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36846us (37145us total) trans 4294972941 rc 0/0 00000100:00100000:1.0:1789760762.188627:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5652 00000100:00100000:1.0:1789760762.188637:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751437448750 time 751482683985 state 0x2 ref 0 00000100:00100000:0.0:1789760762.197920:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760762.198131:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484287232 00000100:00100000:3.0:1789760762.198142:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484287232 00000100:00100000:3.0:1789760762.198211:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751437448750 time 751482683985 state 0x0 ref 1 00000100:00100000:3.0:1789760762.198226:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751482944401 time 751482683985 state 0x1 ref 1 00000100:00100000:3.0:1789760762.198272:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751482944401 time 751529195650 state 0x0 ref 1 00000100:00100000:3.0:1789760762.198287:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484287232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760762.198337:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23035904, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb007 00010000:00010000:3.0:1789760762.198347:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.198364:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:3.0:1789760762.198369:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760762.198375:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23035904->23039999] (req 23035904->23039999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.198389:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23035904->23039999] (req 23035904->23039999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760762.253623:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a32680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508929792:0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.253691:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760762.253748:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508930304:0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.253775:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760762.253797:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508930816:0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.253815:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760762.253834:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1789760762.253839:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508929920:0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.253856:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760762.253874:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508930432:0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.253895:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760762.253911:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b83100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508930944:0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.253928:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760762.253958:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508929792 00000100:00100000:1.0:1789760762.253965:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508929792 00000100:00100000:1.0:1789760762.254003:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 455 00000100:00100000:1.0:1789760762.254012:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b82a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508929792:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.254085:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b82a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508929792:12345-0@lo:400:kworker.0 Request processed in 52us (372us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760762.254093:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 455 00000100:00100000:1.0:1789760762.254116:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508930432 00000100:00100000:1.0:1789760762.254120:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508930432 00000100:00100000:1.0:1789760762.254151:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508930944 00000100:00100000:1.0:1789760762.254154:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508930944 00000100:00100000:1.0:1789760762.254190:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 333 00000100:00100000:1.0:1789760762.254196:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508930432:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.254227:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508930432:12345-0@lo:400:kworker.0 Request processed in 31us (331us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760762.254234:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 333 00000100:00100000:1.0:1789760762.254246:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 334 00000100:00100000:1.0:1789760762.254251:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508930944:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.254277:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b82680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508930944:12345-0@lo:400:kworker.0 Request processed in 26us (347us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760762.254282:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 334 00000100:00080000:1.0:1789760762.254304:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:1.0:1789760762.254308:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1789760762.254339:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508930432:0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.254349:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b83100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508930944:0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.254356:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a32680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508929792:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.254531:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a32d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3480:1876699508929536:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.254564:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760762.254596:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508930048:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.254619:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760762.254635:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508930560:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.254652:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760762.254667:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b82300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508931072:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.254684:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760762.254701:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [3->2] 00000100:00100000:3.0:1789760762.254707:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508929664:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.254727:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760762.254744:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508930176:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.254762:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760762.254775:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508930688:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.254790:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760762.254804:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98dea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508931200:0@lo:13:osp-pre-1-0.0 00000100:00100000:3.0:1789760762.254823:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760762.254851:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508930304 00000100:00100000:3.0:1789760762.254856:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508930304 00000100:00100000:3.0:1789760762.254886:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508930816 00000100:00100000:3.0:1789760762.254890:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508930816 00000100:00100000:3.0:1789760762.254904:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508929920 00000100:00100000:3.0:1789760762.254906:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508929920 00000100:00100000:3.0:1789760762.254918:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508929664 00000100:00100000:3.0:1789760762.254920:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508929664 00000100:00100000:3.0:1789760762.254929:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508930176 00000100:00100000:3.0:1789760762.254931:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508930176 00000100:00100000:3.0:1789760762.254941:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508930688 00000100:00100000:3.0:1789760762.254944:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508930688 00000100:00100000:3.0:1789760762.254958:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 955 00000100:00100000:3.0:1789760762.254964:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508930304:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255003:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3478:x1876699508930304:12345-0@lo:400:kworker.0 Request processed in 40us (1228us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255010:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 955 00000100:00100000:3.0:1789760762.255024:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 956 00000100:00100000:3.0:1789760762.255030:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508930816:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255057:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3478:x1876699508930816:12345-0@lo:400:kworker.0 Request processed in 27us (1240us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255063:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 956 00000100:00100000:3.0:1789760762.255101:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 957 00000100:00100000:3.0:1789760762.255107:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508929920:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255135:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508929920:12345-0@lo:400:kworker.0 Request processed in 29us (1277us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255141:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 957 00000100:00100000:3.0:1789760762.255153:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 958 00000100:00100000:3.0:1789760762.255171:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dc700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508929664:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255202:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dc700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508929664:12345-0@lo:400:kworker.0 Request processed in 43us (473us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255207:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 958 00000100:00100000:3.0:1789760762.255218:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 959 00000100:00100000:3.0:1789760762.255223:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dd180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508930176:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255246:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dd180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508930176:12345-0@lo:400:kworker.0 Request processed in 23us (483us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255251:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 959 00000100:00100000:3.0:1789760762.255260:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 960 00000100:00100000:3.0:1789760762.255277:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dc000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508930688:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255301:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dc000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508930688:12345-0@lo:400:kworker.0 Request processed in 25us (510us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255307:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 960 00000100:00100000:3.0:1789760762.255331:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508930048 00000100:00100000:3.0:1789760762.255335:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508930048 00000100:00100000:3.0:1789760762.255357:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 456 00000100:00100000:3.0:1789760762.255363:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98ddf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508930048:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255391:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98ddf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508930048:12345-0@lo:400:kworker.0 Request processed in 28us (771us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255396:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 456 00000100:00100000:3.0:1789760762.255411:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508930560 00000100:00100000:3.0:1789760762.255414:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508930560 00000100:00100000:3.0:1789760762.255432:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508931072 00000100:00100000:3.0:1789760762.255435:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508931072 00000100:00100000:3.0:1789760762.255449:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 335 00000100:00100000:3.0:1789760762.255455:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508930560:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255480:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508930560:12345-0@lo:400:kworker.0 Request processed in 26us (827us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255486:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 335 00000100:00100000:3.0:1789760762.255496:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 336 00000100:00100000:3.0:1789760762.255501:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98de680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508931072:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255525:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98de680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508931072:12345-0@lo:400:kworker.0 Request processed in 24us (840us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255530:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 336 00000100:00100000:3.0:1789760762.255556:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508931200 00000100:00100000:3.0:1789760762.255560:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508931200 00000100:00100000:3.0:1789760762.255584:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 396 00000100:00100000:3.0:1789760762.255592:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508931200:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:3.0:1789760762.255619:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:3.0:1789760762.255649:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508931200:12345-0@lo:13:osp-pre-1-0.0 Request processed in 57us (824us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760762.255655:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 396 00000100:00080000:3.0:1789760762.255678:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:3.0:1789760762.255682:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:3.0:1789760762.255684:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:3.0:1789760762.255687:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:3.0:1789760762.255717:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a33800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508929664:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255726:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b81180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508930176:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255734:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b82d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508930688:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255749:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98dea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508931200:0@lo:13:osp-pre-1-0.0 00000100:00100000:3.0:1789760762.255759:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b80e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508930048:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255768:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b80380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508930560:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255775:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b82300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508931072:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255813:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508929920:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255823:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b80700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0000_UUID:3478:1876699508930304:0@lo:400:kworker.0 00000100:00100000:3.0:1789760762.255832:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b83b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-lwp-OST0001_UUID:3478:1876699508930816:0@lo:400:kworker.0 00000100:00100000:0.0:1789760762.256027:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760762.256038:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c14380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508931328:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760762.256095:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760762.256123:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508931328 00000100:00100000:0.0:1789760762.256127:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508931328 00000100:00100000:0.0:1789760762.256156:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 397 00000100:00100000:0.0:1789760762.256176:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508931328:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:0.0:1789760762.256188:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:0.0:1789760762.256218:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508931328:12345-0@lo:13:osp-pre-1-1.0 Request processed in 45us (123us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760762.256225:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 397 00000100:00100000:0.0:1789760762.256253:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c14380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508931328:0@lo:13:osp-pre-1-1.0 00000100:00100000:1.0:1789760762.258458:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508929536 00000100:00100000:1.0:1789760762.258471:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508929536 00000100:00100000:1.0:1789760762.258526:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 486 00000100:00100000:1.0:1789760762.258536:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8038850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3480:x1876699508929536:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760762.258624:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8038850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3480:x1876699508929536:12345-0@lo:400:kworker.0 Request processed in 89us (4058us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760762.258632:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 486 00000100:00100000:0.0:1789760762.258732:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a32d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3480:1876699508929536:0@lo:400:kworker.0 00002000:00010000:1.0:1789760762.267318:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23035904 -> 23040000 00010000:00010000:1.0:1789760762.267339:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23035904->23039999] (req 23035904->23039999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.267353:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23035904->23039999] (req 23035904->23039999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.267363:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23035904->23039999] (req 23035904->23039999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.267370:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760762.267375:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23035904->23039999] (req 23035904->23039999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.267384:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23035904->23039999] (req 23035904->23039999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.267406:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23035904->23039999] (req 23035904->23039999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.267414:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209385 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23035904->23039999] (req 23035904->23039999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760762.267515:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484287232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69226us (69600us total) trans 4294972942 rc 0/0 00000100:00100000:1.0:1789760762.267525:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5653 00000100:00100000:1.0:1789760762.267535:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751482944401 time 751529195650 state 0x2 ref 0 00000001:00080000:0.0:1789760762.268349:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972942 is committed 00000100:00100000:3.0:1789760762.270442:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760762.270537:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484287360 00000100:00100000:0.0:1789760762.270546:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484287360 00000100:00100000:0.0:1789760762.270619:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751482944401 time 751529195650 state 0x0 ref 1 00000100:00100000:0.0:1789760762.270636:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751529456066 time 751529195650 state 0x1 ref 1 00000100:00100000:0.0:1789760762.270668:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751529456066 time 751601598545 state 0x0 ref 1 00000100:00100000:0.0:1789760762.270682:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484287360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760762.270746:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23040000, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb007 00010000:00010000:0.0:1789760762.270756:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.270773:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:0.0:1789760762.270778:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760762.270785:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23040000->23044095] (req 23040000->23044095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.270800:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23040000->23044095] (req 23040000->23044095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760762.333924:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972943 is committed 00002000:00010000:2.0:1789760762.334192:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23040000 -> 23044096 00010000:00010000:2.0:1789760762.334207:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23040000->23044095] (req 23040000->23044095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.334219:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23040000->23044095] (req 23040000->23044095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.334228:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23040000->23044095] (req 23040000->23044095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.334235:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760762.334240:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23040000->23044095] (req 23040000->23044095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.334319:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23040000->23044095] (req 23040000->23044095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.334339:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23040000->23044095] (req 23040000->23044095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.334352:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620938c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23040000->23044095] (req 23040000->23044095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760762.334483:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484287360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63800us (64046us total) trans 4294972943 rc 0/0 00000100:00100000:2.0:1789760762.334493:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5654 00000100:00100000:2.0:1789760762.334502:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751529456066 time 751601598545 state 0x2 ref 0 00000100:00100000:3.0:1789760762.339500:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760762.339584:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484287488 00000100:00100000:1.0:1789760762.339595:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484287488 00000100:00100000:1.0:1789760762.339648:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751529456066 time 751601598545 state 0x0 ref 1 00000100:00100000:1.0:1789760762.339663:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751601858961 time 751601598545 state 0x1 ref 1 00000100:00100000:1.0:1789760762.339697:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751601858961 time 751670627449 state 0x0 ref 1 00000100:00100000:1.0:1789760762.339709:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484287488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760762.339771:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23044096, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb007 00010000:00010000:1.0:1789760762.339781:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.339798:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:1.0:1789760762.339803:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760762.339809:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23044096->23048191] (req 23044096->23048191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.339824:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23044096->23048191] (req 23044096->23048191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760762.373306:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23044096 -> 23048192 00010000:00010000:3.0:1789760762.373326:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23044096->23048191] (req 23044096->23048191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.373339:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23044096->23048191] (req 23044096->23048191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.373348:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23044096->23048191] (req 23044096->23048191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.373361:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760762.373366:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23044096->23048191] (req 23044096->23048191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.373375:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23044096->23048191] (req 23044096->23048191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.373395:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23044096->23048191] (req 23044096->23048191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.373403:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209393 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23044096->23048191] (req 23044096->23048191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760762.373493:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484287488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33781us (33999us total) trans 4294972944 rc 0/0 00000100:00100000:3.0:1789760762.373503:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5655 00000100:00100000:3.0:1789760762.373513:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751601858961 time 751670627449 state 0x2 ref 0 00000001:00080000:3.0:1789760762.373550:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972944 is committed 00000100:00100000:0.0:1789760762.384775:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760762.384857:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484287616 00000100:00100000:1.0:1789760762.384867:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484287616 00000100:00100000:1.0:1789760762.384920:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751601858961 time 751670627449 state 0x0 ref 1 00000100:00100000:1.0:1789760762.384936:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751670887865 time 751670627449 state 0x1 ref 1 00000100:00100000:1.0:1789760762.384978:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751670887865 time 751715908993 state 0x0 ref 1 00000100:00100000:1.0:1789760762.384990:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484287616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760762.385038:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23048192, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb007 00010000:00010000:1.0:1789760762.385047:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.385105:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:1.0:1789760762.385110:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760762.385118:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23048192->23052287] (req 23048192->23052287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.385133:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23048192->23052287] (req 23048192->23052287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760762.458481:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972945 is committed 00002000:00010000:3.0:1789760762.458565:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23048192 -> 23052288 00010000:00010000:3.0:1789760762.458579:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23048192->23052287] (req 23048192->23052287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.458593:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23048192->23052287] (req 23048192->23052287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.458608:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23048192->23052287] (req 23048192->23052287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.458614:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760762.458619:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23048192->23052287] (req 23048192->23052287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.458627:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23048192->23052287] (req 23048192->23052287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.458645:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23048192->23052287] (req 23048192->23052287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.458652:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620939a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23048192->23052287] (req 23048192->23052287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760762.458740:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484287616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73748us (73969us total) trans 4294972945 rc 0/0 00000100:00100000:3.0:1789760762.458749:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5656 00000100:00100000:3.0:1789760762.458759:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751670887865 time 751715908993 state 0x2 ref 0 00000100:00100000:0.0:1789760762.462357:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760762.462444:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484287744 00000100:00100000:3.0:1789760762.462453:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484287744 00000100:00100000:3.0:1789760762.462583:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751670887865 time 751715908993 state 0x0 ref 1 00000100:00100000:3.0:1789760762.462599:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751716169409 time 751715908993 state 0x1 ref 1 00000100:00100000:3.0:1789760762.462678:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751716169409 time 751793607366 state 0x0 ref 1 00000100:00100000:3.0:1789760762.462691:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484287744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760762.462738:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23052288, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb007 00010000:00010000:3.0:1789760762.462747:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.462762:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:3.0:1789760762.462766:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760762.462772:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23052288->23056383] (req 23052288->23056383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.462784:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23052288->23056383] (req 23052288->23056383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760762.505719:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972946 is committed 00002000:00010000:3.0:1789760762.506688:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23052288 -> 23056384 00010000:00010000:3.0:1789760762.506705:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23052288->23056383] (req 23052288->23056383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.506718:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23052288->23056383] (req 23052288->23056383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.506726:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23052288->23056383] (req 23052288->23056383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.506732:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760762.506738:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23052288->23056383] (req 23052288->23056383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.506746:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23052288->23056383] (req 23052288->23056383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.506766:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23052288->23056383] (req 23052288->23056383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.506773:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093a1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23052288->23056383] (req 23052288->23056383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760762.506870:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484287744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44177us (44518us total) trans 4294972946 rc 0/0 00000100:00100000:3.0:1789760762.506880:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5657 00000100:00100000:3.0:1789760762.506889:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751716169409 time 751793607366 state 0x2 ref 0 00000100:00100000:2.0:1789760762.513953:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760762.514009:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484287872 00000100:00100000:0.0:1789760762.514020:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484287872 00000100:00100000:0.0:1789760762.514110:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751716169409 time 751793607366 state 0x0 ref 1 00000100:00100000:0.0:1789760762.514128:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751793867782 time 751793607366 state 0x1 ref 1 00000100:00100000:0.0:1789760762.514151:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751793867782 time 751845081845 state 0x0 ref 1 00000100:00100000:0.0:1789760762.514174:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484287872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760762.514229:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23056384, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb007 00010000:00010000:0.0:1789760762.514239:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.514255:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:0.0:1789760762.514259:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760762.514265:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23056384->23060479] (req 23056384->23060479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.514279:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23056384->23060479] (req 23056384->23060479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760762.541693:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972947 is committed 00002000:00010000:2.0:1789760762.541769:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23056384 -> 23060480 00010000:00010000:2.0:1789760762.541783:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23056384->23060479] (req 23056384->23060479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.541796:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23056384->23060479] (req 23056384->23060479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.541806:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23056384->23060479] (req 23056384->23060479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.541814:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760762.541819:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23056384->23060479] (req 23056384->23060479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.541833:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23056384->23060479] (req 23056384->23060479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.541853:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23056384->23060479] (req 23056384->23060479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.541861:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962093a8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23056384->23060479] (req 23056384->23060479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760762.542059:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484287872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27884us (28117us total) trans 4294972947 rc 0/0 00000100:00100000:2.0:1789760762.542114:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5658 00000100:00100000:2.0:1789760762.542123:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751793867782 time 751845081845 state 0x2 ref 0 00000100:00100000:2.0:1789760762.548832:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760762.548916:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484288000 00000100:00100000:1.0:1789760762.548927:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484288000 00000100:00100000:1.0:1789760762.548979:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751793867782 time 751845081845 state 0x0 ref 1 00000100:00100000:1.0:1789760762.548997:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751845342261 time 751845081845 state 0x1 ref 1 00000100:00100000:1.0:1789760762.549026:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751845342261 time 751879957457 state 0x0 ref 1 00000100:00100000:1.0:1789760762.549037:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484288000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760762.549108:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23060480, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb007 00010000:00010000:1.0:1789760762.549117:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.549132:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:1.0:1789760762.549137:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760762.549143:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23060480->23064575] (req 23060480->23064575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.549156:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23060480->23064575] (req 23060480->23064575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760762.574820:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23060480 -> 23064576 00010000:00010000:1.0:1789760762.574840:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23060480->23064575] (req 23060480->23064575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.574853:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23060480->23064575] (req 23060480->23064575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.574863:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23060480->23064575] (req 23060480->23064575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.574876:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760762.574881:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23060480->23064575] (req 23060480->23064575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.574891:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23060480->23064575] (req 23060480->23064575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.574911:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23060480->23064575] (req 23060480->23064575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.574919:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093af lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23060480->23064575] (req 23060480->23064575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760762.575134:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484288000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26093us (26303us total) trans 4294972948 rc 0/0 00000100:00100000:1.0:1789760762.575144:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5659 00000100:00100000:1.0:1789760762.575153:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751845342261 time 751879957457 state 0x2 ref 0 00000001:00080000:1.0:1789760762.575200:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972948 is committed 00000100:00100000:2.0:1789760762.578840:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760762.578952:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484288128 00000100:00100000:3.0:1789760762.578961:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484288128 00000100:00100000:3.0:1789760762.579015:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751845342261 time 751879957457 state 0x0 ref 1 00000100:00100000:3.0:1789760762.579030:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751880217873 time 751879957457 state 0x1 ref 1 00000100:00100000:0.0:1789760762.582957:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751880217873 time 751913876605 state 0x0 ref 1 00000100:00100000:0.0:1789760762.582995:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484288128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760762.583050:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23064576, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb007 00010000:00010000:0.0:1789760762.583060:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.584357:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:0.0:1789760762.584362:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760762.584370:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23064576->23068671] (req 23064576->23068671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.584385:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23064576->23068671] (req 23064576->23068671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760762.665176:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972949 is committed 00002000:00010000:0.0:1789760762.665674:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23064576 -> 23068672 00010000:00010000:0.0:1789760762.665688:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23064576->23068671] (req 23064576->23068671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.665702:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23064576->23068671] (req 23064576->23068671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.665710:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23064576->23068671] (req 23064576->23068671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.665717:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760762.665722:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23064576->23068671] (req 23064576->23068671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.665731:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23064576->23068671] (req 23064576->23068671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.665749:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23064576->23068671] (req 23064576->23068671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.665756:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962093b6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23064576->23068671] (req 23064576->23068671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760762.665887:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484288128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 82905us (87052us total) trans 4294972949 rc 0/0 00000100:00100000:0.0:1789760762.665898:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5660 00000100:00100000:0.0:1789760762.665907:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751880217873 time 751913876605 state 0x2 ref 0 00000100:00100000:2.0:1789760762.669415:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760762.669573:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484288256 00000100:00100000:3.0:1789760762.669582:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484288256 00000100:00100000:3.0:1789760762.669634:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751880217873 time 751913876605 state 0x0 ref 1 00000100:00100000:3.0:1789760762.669649:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751914137021 time 751913876605 state 0x1 ref 1 00000100:00100000:3.0:1789760762.669680:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751914137021 time 752000611212 state 0x0 ref 1 00000100:00100000:3.0:1789760762.669693:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484288256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760762.669741:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23068672, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb007 00010000:00010000:3.0:1789760762.669750:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.669767:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=751s v=(0 0) 00010000:00010000:3.0:1789760762.669771:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760762.669778:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23068672->23072767] (req 23068672->23072767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.669791:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23068672->23072767] (req 23068672->23072767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760762.752106:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972950 is committed 00002000:00010000:3.0:1789760762.755422:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23068672 -> 23072768 00002000:00010000:3.0:1789760762.755431:0:8106:0:(ofd_lvb.c:331:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb blocks from disk: 45063 -> 47111 00010000:00010000:3.0:1789760762.755505:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23068672->23072767] (req 23068672->23072767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.755520:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23068672->23072767] (req 23068672->23072767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.755530:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23068672->23072767] (req 23068672->23072767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.755543:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760762.755548:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23068672->23072767] (req 23068672->23072767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.755557:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23068672->23072767] (req 23068672->23072767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.755580:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23068672->23072767] (req 23068672->23072767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.755587:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093bd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23068672->23072767] (req 23068672->23072767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760762.755725:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484288256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 86029us (86314us total) trans 4294972950 rc 0/0 00000100:00100000:3.0:1789760762.755735:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5661 00000100:00100000:3.0:1789760762.755744:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751914137021 time 752000611212 state 0x2 ref 0 00000100:00100000:0.0:1789760762.760789:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760762.760867:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484288384 00000100:00100000:2.0:1789760762.760877:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484288384 00000100:00100000:2.0:1789760762.760924:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 751914137021 time 752000611212 state 0x0 ref 1 00000100:00100000:2.0:1789760762.760954:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752000871628 time 752000611212 state 0x1 ref 1 00000100:00100000:2.0:1789760762.760979:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752000871628 time 752091910907 state 0x0 ref 1 00000100:00100000:2.0:1789760762.760992:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484288384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760762.761037:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23072768, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb807 00010000:00010000:2.0:1789760762.761045:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.761060:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:2.0:1789760762.761093:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760762.761100:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23072768->23076863] (req 23072768->23076863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.761113:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23072768->23076863] (req 23072768->23076863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.764619:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760762.813257:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972951 is committed 00002000:00010000:2.0:1789760762.813419:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23072768 -> 23076864 00010000:00010000:2.0:1789760762.813434:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23072768->23076863] (req 23072768->23076863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.813448:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23072768->23076863] (req 23072768->23076863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.813458:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23072768->23076863] (req 23072768->23076863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.813465:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760762.813471:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23072768->23076863] (req 23072768->23076863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.813481:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23072768->23076863] (req 23072768->23076863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.813502:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23072768->23076863] (req 23072768->23076863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.813510:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093c4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23072768->23076863] (req 23072768->23076863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760762.813641:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484288384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52647us (52856us total) trans 4294972951 rc 0/0 00000100:00100000:2.0:1789760762.813653:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5662 00000100:00100000:2.0:1789760762.813663:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752000871628 time 752091910907 state 0x2 ref 0 00000100:00100000:3.0:1789760762.816819:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760762.817254:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484288512 00000100:00100000:1.0:1789760762.817263:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484288512 00000100:00100000:1.0:1789760762.817313:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752000871628 time 752091910907 state 0x0 ref 1 00000100:00100000:1.0:1789760762.817328:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752092171323 time 752091910907 state 0x1 ref 1 00000100:00100000:1.0:1789760762.817363:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752092171323 time 752148294468 state 0x0 ref 1 00000100:00100000:1.0:1789760762.817375:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484288512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760762.817423:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23076864, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb807 00010000:00010000:1.0:1789760762.817432:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.817448:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:1.0:1789760762.817453:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760762.817459:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23076864->23080959] (req 23076864->23080959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760762.817473:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23076864->23080959] (req 23076864->23080959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760762.852003:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972952 is committed 00002000:00010000:2.0:1789760762.852227:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23076864 -> 23080960 00010000:00010000:2.0:1789760762.852241:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23076864->23080959] (req 23076864->23080959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.852254:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23076864->23080959] (req 23076864->23080959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.852264:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23076864->23080959] (req 23076864->23080959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.852271:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760762.852276:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23076864->23080959] (req 23076864->23080959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.852289:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23076864->23080959] (req 23076864->23080959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.852308:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23076864->23080959] (req 23076864->23080959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.852316:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962093cb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23076864->23080959] (req 23076864->23080959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760762.852391:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484288512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35015us (35577us total) trans 4294972952 rc 0/0 00000100:00100000:2.0:1789760762.852398:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5663 00000100:00100000:2.0:1789760762.852408:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752092171323 time 752148294468 state 0x2 ref 0 00000100:00100000:3.0:1789760762.855411:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760762.855493:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484288640 00000100:00100000:0.0:1789760762.855503:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484288640 00000100:00100000:0.0:1789760762.855565:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752092171323 time 752148294468 state 0x0 ref 1 00000100:00100000:0.0:1789760762.855580:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752148554884 time 752148294468 state 0x1 ref 1 00000100:00100000:0.0:1789760762.855628:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752148554884 time 752186550141 state 0x0 ref 1 00000100:00100000:0.0:1789760762.855641:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484288640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760762.855694:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23080960, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb807 00010000:00010000:0.0:1789760762.855704:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.855720:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:0.0:1789760762.855724:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760762.855729:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23080960->23085055] (req 23080960->23085055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.855744:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23080960->23085055] (req 23080960->23085055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760762.889913:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972953 is committed 00002000:00010000:2.0:1789760762.890282:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23080960 -> 23085056 00010000:00010000:2.0:1789760762.890297:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23080960->23085055] (req 23080960->23085055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.890309:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23080960->23085055] (req 23080960->23085055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.890317:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23080960->23085055] (req 23080960->23085055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.890323:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760762.890329:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23080960->23085055] (req 23080960->23085055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.890338:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23080960->23085055] (req 23080960->23085055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.890367:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23080960->23085055] (req 23080960->23085055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.890374:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962093d2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23080960->23085055] (req 23080960->23085055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760762.890909:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484288640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35264us (35500us total) trans 4294972953 rc 0/0 00000100:00100000:2.0:1789760762.890921:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5664 00000100:00100000:2.0:1789760762.890932:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752148554884 time 752186550141 state 0x2 ref 0 00000100:00100000:0.0:1789760762.896490:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760762.896569:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484288768 00000100:00100000:3.0:1789760762.896579:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484288768 00000100:00100000:3.0:1789760762.896635:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752148554884 time 752186550141 state 0x0 ref 1 00000100:00100000:3.0:1789760762.896652:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752186810557 time 752186550141 state 0x1 ref 1 00000100:00100000:3.0:1789760762.896682:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752186810557 time 752227612864 state 0x0 ref 1 00000100:00100000:3.0:1789760762.896696:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484288768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760762.896743:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23085056, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb807 00010000:00010000:3.0:1789760762.896751:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.896767:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:3.0:1789760762.896772:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760762.896778:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23085056->23089151] (req 23085056->23089151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.896792:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23085056->23089151] (req 23085056->23089151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760762.943605:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972954 is committed 00002000:00010000:0.0:1789760762.943688:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23085056 -> 23089152 00010000:00010000:0.0:1789760762.943703:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23085056->23089151] (req 23085056->23089151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.943716:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23085056->23089151] (req 23085056->23089151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.943724:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23085056->23089151] (req 23085056->23089151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.943733:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760762.943737:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23085056->23089151] (req 23085056->23089151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.943751:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23085056->23089151] (req 23085056->23089151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.943769:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23085056->23089151] (req 23085056->23089151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760762.943777:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962093d9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23085056->23089151] (req 23085056->23089151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760762.943900:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484288768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47202us (47414us total) trans 4294972954 rc 0/0 00000100:00100000:0.0:1789760762.943910:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5665 00000100:00100000:0.0:1789760762.943919:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752186810557 time 752227612864 state 0x2 ref 0 00000100:00100000:3.0:1789760762.953824:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760762.954016:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484288896 00000100:00100000:2.0:1789760762.954026:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484288896 00000100:00100000:2.0:1789760762.954123:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752186810557 time 752227612864 state 0x0 ref 1 00000100:00100000:2.0:1789760762.954140:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752227873280 time 752227612864 state 0x1 ref 1 00000100:00100000:2.0:1789760762.954172:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752227873280 time 752285103122 state 0x0 ref 1 00000100:00100000:2.0:1789760762.954186:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484288896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760762.954247:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23089152, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb807 00010000:00010000:2.0:1789760762.954257:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.954272:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:2.0:1789760762.954276:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760762.954282:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23089152->23093247] (req 23089152->23093247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.954295:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23089152->23093247] (req 23089152->23093247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760762.986327:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972955 is committed 00002000:00010000:3.0:1789760762.986572:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23089152 -> 23093248 00010000:00010000:3.0:1789760762.986585:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23089152->23093247] (req 23089152->23093247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.986598:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23089152->23093247] (req 23089152->23093247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.986607:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23089152->23093247] (req 23089152->23093247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.986614:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760762.986618:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23089152->23093247] (req 23089152->23093247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.986628:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23089152->23093247] (req 23089152->23093247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.986647:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23089152->23093247] (req 23089152->23093247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760762.986660:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962093e0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23089152->23093247] (req 23089152->23093247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760762.986751:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484288896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32564us (32932us total) trans 4294972955 rc 0/0 00000100:00100000:3.0:1789760762.986761:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5666 00000100:00100000:3.0:1789760762.986771:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752227873280 time 752285103122 state 0x2 ref 0 00000100:00100000:3.0:1789760762.991325:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760762.991468:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484289024 00000100:00100000:2.0:1789760762.991479:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484289024 00000100:00100000:2.0:1789760762.991532:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752227873280 time 752285103122 state 0x0 ref 1 00000100:00100000:2.0:1789760762.991548:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752285363538 time 752285103122 state 0x1 ref 1 00000100:00100000:2.0:1789760762.991579:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752285363538 time 752322509905 state 0x0 ref 1 00000100:00100000:2.0:1789760762.991656:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484289024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760762.991708:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23093248, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb807 00010000:00010000:2.0:1789760762.991717:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.991738:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:2.0:1789760762.991742:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760762.991749:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23093248->23097343] (req 23093248->23097343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760762.991761:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23093248->23097343] (req 23093248->23097343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760763.084951:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972956 is committed 00002000:00010000:2.0:1789760763.085015:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23093248 -> 23097344 00010000:00010000:2.0:1789760763.085029:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23093248->23097343] (req 23093248->23097343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.085051:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23093248->23097343] (req 23093248->23097343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.085060:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23093248->23097343] (req 23093248->23097343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.085100:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760763.085105:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23093248->23097343] (req 23093248->23097343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.085114:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23093248->23097343] (req 23093248->23097343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.085139:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23093248->23097343] (req 23093248->23097343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.085146:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093e7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23093248->23097343] (req 23093248->23097343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760763.085252:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484289024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 93657us (93931us total) trans 4294972956 rc 0/0 00000100:00100000:2.0:1789760763.085261:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5667 00000100:00100000:2.0:1789760763.085271:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752285363538 time 752322509905 state 0x2 ref 0 00000100:00100000:0.0:1789760763.095148:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760763.095351:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484289152 00000100:00100000:2.0:1789760763.095361:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484289152 00000100:00100000:2.0:1789760763.095413:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752285363538 time 752322509905 state 0x0 ref 1 00000100:00100000:2.0:1789760763.095428:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752322770321 time 752322509905 state 0x1 ref 1 00000100:00100000:2.0:1789760763.095468:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752322770321 time 752426398177 state 0x0 ref 1 00000100:00100000:2.0:1789760763.095480:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706c450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484289152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760763.095526:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23097344, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb807 00010000:00010000:2.0:1789760763.095535:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.095550:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:2.0:1789760763.095556:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760763.095562:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23097344->23101439] (req 23097344->23101439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.095576:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23097344->23101439] (req 23097344->23101439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760763.111209:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760763.111251:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760763.111282:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760763.111302:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.111310:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484289408 00000100:00100000:3.0:1789760763.111321:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484289408 00000100:00100000:0.0:1789760763.111372:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.111373:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484289664 00000100:00100000:3.0:1789760763.111376:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484289664 00000100:00100000:3.0:1789760763.111396:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484289536 00000100:00100000:3.0:1789760763.111399:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484289536 00000100:00100000:3.0:1789760763.111413:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484289792 00000100:00100000:3.0:1789760763.111416:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484289792 00000100:00100000:0.0:1789760763.111419:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 961 00000100:00100000:3.0:1789760763.111431:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 962 00000100:00100000:0.0:1789760763.111435:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484289408:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760763.111437:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484289664:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760763.111502:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484289408:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 73us (249us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760763.111510:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 961 00000100:00100000:3.0:1789760763.111519:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c10e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484289664:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 82us (238us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760763.111525:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 962 00000100:00100000:0.0:1789760763.111529:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 963 00000100:00100000:0.0:1789760763.111537:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484289536:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760763.111539:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 964 00000100:00100000:3.0:1789760763.111546:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb5880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484289792:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760763.111564:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484289536:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 29us (261us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760763.111565:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb5880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2425:x1876699484289792:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 21us (194us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760763.111571:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 963 00000100:00100000:1.0:1789760763.111627:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484289280 00000100:00100000:1.0:1789760763.111636:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484289280 00000100:00100000:1.0:1789760763.111666:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 487 00000100:00100000:1.0:1789760763.111672:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c803a850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2425:x1876699484289280:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760763.111695:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c803a850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2425:x1876699484289280:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 24us (507us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760763.111701:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 487 00000100:00080000:2.0:1789760763.113685:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760763.113696:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:3.0:1789760763.115416:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 964 00000001:00080000:1.0:1789760763.142091:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972957 is committed 00002000:00010000:3.0:1789760763.142355:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23097344 -> 23101440 00010000:00010000:3.0:1789760763.142372:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23097344->23101439] (req 23097344->23101439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.142386:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23097344->23101439] (req 23097344->23101439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.142396:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23097344->23101439] (req 23097344->23101439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.142404:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760763.142408:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23097344->23101439] (req 23097344->23101439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.142417:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23097344->23101439] (req 23097344->23101439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.142435:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23097344->23101439] (req 23097344->23101439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.142442:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962093ee lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23097344->23101439] (req 23097344->23101439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760763.143365:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706c450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484289152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47878us (48218us total) trans 4294972957 rc 0/0 00000100:00100000:3.0:1789760763.143377:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5668 00000100:00100000:3.0:1789760763.143387:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752322770321 time 752426398177 state 0x2 ref 0 00000100:00100000:0.0:1789760763.151187:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.151349:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484289920 00000100:00100000:3.0:1789760763.151359:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484289920 00000100:00100000:3.0:1789760763.151411:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752322770321 time 752426398177 state 0x0 ref 1 00000100:00100000:3.0:1789760763.151426:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752426658593 time 752426398177 state 0x1 ref 1 00000100:00100000:3.0:1789760763.151455:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752426658593 time 752482386379 state 0x0 ref 1 00000100:00100000:3.0:1789760763.151467:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484289920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760763.151515:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23101440, mtime: 0x6aad94fa, atime: 0x0, ctime: 0x6aad94fa, blocks: 0xb807 00010000:00010000:3.0:1789760763.151524:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.151539:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:3.0:1789760763.151544:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760763.151550:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23101440->23105535] (req 23101440->23105535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.151563:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23101440->23105535] (req 23101440->23105535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760763.203569:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972958 is committed 00002000:00010000:1.0:1789760763.203687:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23101440 -> 23105536 00002000:00010000:1.0:1789760763.203692:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760762 -> 1789760763 00002000:00010000:1.0:1789760763.203695:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760762 -> 1789760763 00010000:00010000:1.0:1789760763.203709:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23101440->23105535] (req 23101440->23105535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.203722:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23101440->23105535] (req 23101440->23105535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.203731:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23101440->23105535] (req 23101440->23105535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.203739:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760763.203743:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23101440->23105535] (req 23101440->23105535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.203753:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23101440->23105535] (req 23101440->23105535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.203772:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23101440->23105535] (req 23101440->23105535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.203779:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962093f5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23101440->23105535] (req 23101440->23105535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760763.203872:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484289920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52402us (52690us total) trans 4294972958 rc 0/0 00000100:00100000:1.0:1789760763.203883:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5669 00000100:00100000:1.0:1789760763.203893:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752426658593 time 752482386379 state 0x2 ref 0 00000100:00100000:2.0:1789760763.208230:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.208385:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484290048 00000100:00100000:3.0:1789760763.208396:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484290048 00000100:00100000:3.0:1789760763.208449:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752426658593 time 752482386379 state 0x0 ref 1 00000100:00100000:3.0:1789760763.208464:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752482646795 time 752482386379 state 0x1 ref 1 00000100:00100000:3.0:1789760763.208494:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752482646795 time 752539425274 state 0x0 ref 1 00000100:00100000:3.0:1789760763.208505:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484290048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760763.208552:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23105536, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:3.0:1789760763.208560:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.208575:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:3.0:1789760763.208579:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760763.208585:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23105536->23109631] (req 23105536->23109631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.208598:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23105536->23109631] (req 23105536->23109631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760763.266446:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972959 is committed 00002000:00010000:3.0:1789760763.266549:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23105536 -> 23109632 00010000:00010000:3.0:1789760763.266566:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23105536->23109631] (req 23105536->23109631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.266578:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23105536->23109631] (req 23105536->23109631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.266587:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23105536->23109631] (req 23105536->23109631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.266593:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760763.266598:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23105536->23109631] (req 23105536->23109631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.266607:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23105536->23109631] (req 23105536->23109631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.266627:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23105536->23109631] (req 23105536->23109631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.266634:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962093fc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23105536->23109631] (req 23105536->23109631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760763.266731:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484290048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58222us (58505us total) trans 4294972959 rc 0/0 00000100:00100000:3.0:1789760763.266745:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5670 00000100:00100000:3.0:1789760763.266754:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752482646795 time 752539425274 state 0x2 ref 0 00000100:00100000:2.0:1789760763.270769:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760763.270848:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484290176 00000100:00100000:1.0:1789760763.270857:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484290176 00000100:00100000:1.0:1789760763.270911:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752482646795 time 752539425274 state 0x0 ref 1 00000100:00100000:1.0:1789760763.270926:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752539685690 time 752539425274 state 0x1 ref 1 00000100:00100000:1.0:1789760763.271042:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752539685690 time 752601971617 state 0x0 ref 1 00000100:00100000:1.0:1789760763.271056:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484290176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760763.271123:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23109632, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:1.0:1789760763.271132:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.271148:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:1.0:1789760763.271163:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760763.271170:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23109632->23113727] (req 23109632->23113727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.271184:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23109632->23113727] (req 23109632->23113727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760763.303374:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23109632 -> 23113728 00010000:00010000:1.0:1789760763.303398:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23109632->23113727] (req 23109632->23113727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.303412:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23109632->23113727] (req 23109632->23113727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.303422:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23109632->23113727] (req 23109632->23113727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.303430:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760763.303435:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23109632->23113727] (req 23109632->23113727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.303444:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23109632->23113727] (req 23109632->23113727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.303464:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23109632->23113727] (req 23109632->23113727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.303472:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209403 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23109632->23113727] (req 23109632->23113727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760763.303581:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484290176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32518us (32812us total) trans 4294972960 rc 0/0 00000100:00100000:1.0:1789760763.303592:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5671 00000100:00100000:1.0:1789760763.303603:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752539685690 time 752601971617 state 0x2 ref 0 00000001:00080000:2.0:1789760763.303682:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972960 is committed 00000100:00100000:1.0:1789760763.307567:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760763.307662:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484290304 00000100:00100000:2.0:1789760763.307673:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484290304 00000100:00100000:2.0:1789760763.307727:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752539685690 time 752601971617 state 0x0 ref 1 00000100:00100000:2.0:1789760763.307743:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752602232033 time 752601971617 state 0x1 ref 1 00000100:00100000:2.0:1789760763.307810:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752602232033 time 752638740429 state 0x0 ref 1 00000100:00100000:2.0:1789760763.307823:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484290304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760763.307880:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23113728, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:2.0:1789760763.307888:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.307903:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:2.0:1789760763.307908:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760763.307914:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23113728->23117823] (req 23113728->23117823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.307927:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23113728->23117823] (req 23113728->23117823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760763.337595:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972961 is committed 00002000:00010000:2.0:1789760763.337698:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23113728 -> 23117824 00010000:00010000:2.0:1789760763.337714:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23113728->23117823] (req 23113728->23117823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.337727:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23113728->23117823] (req 23113728->23117823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.337736:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23113728->23117823] (req 23113728->23117823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.337743:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760763.337748:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23113728->23117823] (req 23113728->23117823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.337758:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23113728->23117823] (req 23113728->23117823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.337778:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23113728->23117823] (req 23113728->23117823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.337786:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620940a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23113728->23117823] (req 23113728->23117823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760763.337884:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484290304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30058us (30321us total) trans 4294972961 rc 0/0 00000100:00100000:2.0:1789760763.337894:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5672 00000100:00100000:2.0:1789760763.337904:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752602232033 time 752638740429 state 0x2 ref 0 00000100:00100000:2.0:1789760763.341405:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760763.341590:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484290432 00000100:00100000:1.0:1789760763.341599:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484290432 00000100:00100000:1.0:1789760763.341661:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752602232033 time 752638740429 state 0x0 ref 1 00000100:00100000:1.0:1789760763.341676:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752639000845 time 752638740429 state 0x1 ref 1 00000100:00100000:1.0:1789760763.341729:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752639000845 time 752672660222 state 0x0 ref 1 00000100:00100000:1.0:1789760763.341743:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484290432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760763.341787:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23117824, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:1.0:1789760763.341796:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.341812:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:1.0:1789760763.341817:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760763.341823:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23117824->23121919] (req 23117824->23121919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.341837:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23117824->23121919] (req 23117824->23121919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760763.372855:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972962 is committed 00002000:00010000:2.0:1789760763.373024:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23117824 -> 23121920 00010000:00010000:2.0:1789760763.373039:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23117824->23121919] (req 23117824->23121919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.373051:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23117824->23121919] (req 23117824->23121919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.373061:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23117824->23121919] (req 23117824->23121919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.373133:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760763.373139:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23117824->23121919] (req 23117824->23121919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.373148:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23117824->23121919] (req 23117824->23121919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.373180:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23117824->23121919] (req 23117824->23121919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.373194:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209411 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23117824->23121919] (req 23117824->23121919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760763.374895:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484290432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33142us (33487us total) trans 4294972962 rc 0/0 00000100:00100000:2.0:1789760763.374911:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5673 00000100:00100000:2.0:1789760763.374921:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752639000845 time 752672660222 state 0x2 ref 0 00000100:00100000:2.0:1789760763.379386:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.379438:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484290560 00000100:00100000:3.0:1789760763.379447:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484290560 00000100:00100000:3.0:1789760763.379492:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752639000845 time 752672660222 state 0x0 ref 1 00000100:00100000:3.0:1789760763.379507:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752672920638 time 752672660222 state 0x1 ref 1 00000100:00100000:3.0:1789760763.379537:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752672920638 time 752710468574 state 0x0 ref 1 00000100:00100000:3.0:1789760763.379549:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484290560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760763.379594:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23121920, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:3.0:1789760763.379602:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.379618:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:3.0:1789760763.379622:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760763.379628:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23121920->23126015] (req 23121920->23126015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.379639:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23121920->23126015] (req 23121920->23126015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760763.412610:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23121920 -> 23126016 00010000:00010000:3.0:1789760763.412629:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23121920->23126015] (req 23121920->23126015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.412643:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23121920->23126015] (req 23121920->23126015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.412653:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23121920->23126015] (req 23121920->23126015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.412660:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760763.412665:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23121920->23126015] (req 23121920->23126015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.412674:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23121920->23126015] (req 23121920->23126015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.412694:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23121920->23126015] (req 23121920->23126015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.412702:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209418 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23121920->23126015] (req 23121920->23126015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760763.412797:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484290560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33245us (33415us total) trans 4294972963 rc 0/0 00000100:00100000:3.0:1789760763.412807:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5674 00000100:00100000:3.0:1789760763.412817:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752672920638 time 752710468574 state 0x2 ref 0 00000001:00080000:0.0:1789760763.414228:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972963 is committed 00000100:00100000:1.0:1789760763.415500:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760763.415655:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484290688 00000100:00100000:2.0:1789760763.415664:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484290688 00000100:00100000:2.0:1789760763.415714:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752672920638 time 752710468574 state 0x0 ref 1 00000100:00100000:2.0:1789760763.415728:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752710728990 time 752710468574 state 0x1 ref 1 00000100:00100000:2.0:1789760763.415759:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752710728990 time 752746690523 state 0x0 ref 1 00000100:00100000:2.0:1789760763.415772:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484290688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760763.415816:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23126016, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:2.0:1789760763.415824:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.415838:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:2.0:1789760763.415843:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760763.415849:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23126016->23130111] (req 23126016->23130111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.415862:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23126016->23130111] (req 23126016->23130111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760763.446093:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972964 is committed 00002000:00010000:3.0:1789760763.446207:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23126016 -> 23130112 00010000:00010000:3.0:1789760763.446223:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23126016->23130111] (req 23126016->23130111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.446235:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23126016->23130111] (req 23126016->23130111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.446244:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23126016->23130111] (req 23126016->23130111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.446251:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760763.446256:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23126016->23130111] (req 23126016->23130111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.446264:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23126016->23130111] (req 23126016->23130111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.446285:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23126016->23130111] (req 23126016->23130111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.446293:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620941f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23126016->23130111] (req 23126016->23130111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760763.446424:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484290688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30649us (30926us total) trans 4294972964 rc 0/0 00000100:00100000:3.0:1789760763.446434:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5675 00000100:00100000:3.0:1789760763.446507:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752710728990 time 752746690523 state 0x2 ref 0 00000100:00100000:1.0:1789760763.452366:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760763.452463:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484290816 00000100:00100000:2.0:1789760763.452473:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484290816 00000100:00100000:2.0:1789760763.452529:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752710728990 time 752746690523 state 0x0 ref 1 00000100:00100000:2.0:1789760763.452544:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752746950939 time 752746690523 state 0x1 ref 1 00000100:00100000:2.0:1789760763.452588:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752746950939 time 752783510355 state 0x0 ref 1 00000100:00100000:2.0:1789760763.452601:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484290816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760763.452647:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23130112, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:2.0:1789760763.452656:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.452671:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:2.0:1789760763.452689:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760763.452695:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23130112->23134207] (req 23130112->23134207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.452709:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23130112->23134207] (req 23130112->23134207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760763.480470:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23130112 -> 23134208 00010000:00010000:3.0:1789760763.480491:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23130112->23134207] (req 23130112->23134207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.480505:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23130112->23134207] (req 23130112->23134207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.480515:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23130112->23134207] (req 23130112->23134207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.480523:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760763.480527:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23130112->23134207] (req 23130112->23134207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.480536:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23130112->23134207] (req 23130112->23134207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.480556:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23130112->23134207] (req 23130112->23134207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.480564:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209426 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23130112->23134207] (req 23130112->23134207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760763.480658:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484290816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28055us (28296us total) trans 4294972965 rc 0/0 00000100:00100000:3.0:1789760763.480668:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5676 00000100:00100000:3.0:1789760763.480679:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752746950939 time 752783510355 state 0x2 ref 0 00000001:00080000:0.0:1789760763.481722:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972965 is committed 00000100:00100000:2.0:1789760763.485517:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760763.485609:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484290944 00000100:00100000:1.0:1789760763.485619:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484290944 00000100:00100000:1.0:1789760763.485676:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752746950939 time 752783510355 state 0x0 ref 1 00000100:00100000:1.0:1789760763.485691:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752783770771 time 752783510355 state 0x1 ref 1 00000100:00100000:1.0:1789760763.485722:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752783770771 time 752816652556 state 0x0 ref 1 00000100:00100000:1.0:1789760763.485735:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484290944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760763.485782:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23134208, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:1.0:1789760763.485791:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.485808:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:1.0:1789760763.485812:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760763.485819:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23134208->23138303] (req 23134208->23138303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.485832:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23134208->23138303] (req 23134208->23138303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760763.516410:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972966 is committed 00002000:00010000:1.0:1789760763.516646:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23134208 -> 23138304 00010000:00010000:1.0:1789760763.516661:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23134208->23138303] (req 23134208->23138303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.516675:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23134208->23138303] (req 23134208->23138303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.516684:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23134208->23138303] (req 23134208->23138303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.516692:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760763.516697:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23134208->23138303] (req 23134208->23138303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.516707:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23134208->23138303] (req 23134208->23138303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.516728:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23134208->23138303] (req 23134208->23138303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.516736:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620942d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23134208->23138303] (req 23134208->23138303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760763.516829:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484290944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31092us (31323us total) trans 4294972966 rc 0/0 00000100:00100000:1.0:1789760763.516839:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5677 00000100:00100000:1.0:1789760763.516849:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752783770771 time 752816652556 state 0x2 ref 0 00000100:00100000:1.0:1789760763.520584:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.520684:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484291072 00000100:00100000:3.0:1789760763.520695:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484291072 00000100:00100000:3.0:1789760763.520751:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752783770771 time 752816652556 state 0x0 ref 1 00000100:00100000:3.0:1789760763.520766:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752816912972 time 752816652556 state 0x1 ref 1 00000100:00100000:3.0:1789760763.520843:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752816912972 time 752851772656 state 0x0 ref 1 00000100:00100000:3.0:1789760763.520856:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484291072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760763.520905:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23138304, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:3.0:1789760763.520913:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.520928:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:3.0:1789760763.520933:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760763.520940:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23138304->23142399] (req 23138304->23142399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.520954:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23138304->23142399] (req 23138304->23142399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760763.564269:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23138304 -> 23142400 00010000:00010000:1.0:1789760763.564289:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23138304->23142399] (req 23138304->23142399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.564303:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23138304->23142399] (req 23138304->23142399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.564312:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23138304->23142399] (req 23138304->23142399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.564324:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760763.564329:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23138304->23142399] (req 23138304->23142399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.564338:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23138304->23142399] (req 23138304->23142399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.564357:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23138304->23142399] (req 23138304->23142399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.564365:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209434 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23138304->23142399] (req 23138304->23142399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760763.564461:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484291072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43602us (43882us total) trans 4294972967 rc 0/0 00000100:00100000:1.0:1789760763.564471:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5678 00000100:00100000:1.0:1789760763.564482:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752816912972 time 752851772656 state 0x2 ref 0 00000001:00080000:2.0:1789760763.565746:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972967 is committed 00000100:00100000:2.0:1789760763.569691:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.569773:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484291200 00000100:00100000:3.0:1789760763.569782:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484291200 00000100:00100000:3.0:1789760763.569834:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752816912972 time 752851772656 state 0x0 ref 1 00000100:00100000:3.0:1789760763.569853:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752852033072 time 752851772656 state 0x1 ref 1 00000100:00100000:3.0:1789760763.569881:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752852033072 time 752900812091 state 0x0 ref 1 00000100:00100000:3.0:1789760763.569894:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484291200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760763.569952:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23142400, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:3.0:1789760763.569961:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.569976:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:3.0:1789760763.569980:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760763.569986:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23142400->23146495] (req 23142400->23146495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.570000:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23142400->23146495] (req 23142400->23146495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760763.605262:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972968 is committed 00002000:00010000:2.0:1789760763.605361:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23142400 -> 23146496 00010000:00010000:2.0:1789760763.605374:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23142400->23146495] (req 23142400->23146495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.605387:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23142400->23146495] (req 23142400->23146495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.605395:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23142400->23146495] (req 23142400->23146495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.605402:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760763.605406:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23142400->23146495] (req 23142400->23146495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.605414:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23142400->23146495] (req 23142400->23146495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.605434:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23142400->23146495] (req 23142400->23146495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.605453:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620943b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23142400->23146495] (req 23142400->23146495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760763.605550:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484291200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35654us (35864us total) trans 4294972968 rc 0/0 00000100:00100000:2.0:1789760763.605563:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5679 00000100:00100000:2.0:1789760763.605573:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752852033072 time 752900812091 state 0x2 ref 0 00000100:00100000:0.0:1789760763.618241:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760763.618336:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484291328 00000100:00100000:2.0:1789760763.618345:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484291328 00000100:00100000:2.0:1789760763.618400:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752852033072 time 752900812091 state 0x0 ref 1 00000100:00100000:2.0:1789760763.618415:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752901072507 time 752900812091 state 0x1 ref 1 00000100:00100000:2.0:1789760763.618492:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752901072507 time 752949422245 state 0x0 ref 1 00000100:00100000:2.0:1789760763.618505:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484291328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760763.618554:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23146496, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:2.0:1789760763.618563:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.618578:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:2.0:1789760763.618582:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760763.618588:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23146496->23150591] (req 23146496->23150591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.618602:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23146496->23150591] (req 23146496->23150591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760763.653672:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972969 is committed 00002000:00010000:2.0:1789760763.653779:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23146496 -> 23150592 00010000:00010000:2.0:1789760763.653795:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23146496->23150591] (req 23146496->23150591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.653808:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23146496->23150591] (req 23146496->23150591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.653817:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23146496->23150591] (req 23146496->23150591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.653823:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760763.653828:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23146496->23150591] (req 23146496->23150591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.653836:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23146496->23150591] (req 23146496->23150591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.653857:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23146496->23150591] (req 23146496->23150591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.653864:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209442 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23146496->23150591] (req 23146496->23150591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760763.654004:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484291328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35496us (35767us total) trans 4294972969 rc 0/0 00000100:00100000:2.0:1789760763.654016:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5680 00000100:00100000:2.0:1789760763.654024:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752901072507 time 752949422245 state 0x2 ref 0 00000100:00100000:1.0:1789760763.661148:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760763.661285:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484291456 00000100:00100000:2.0:1789760763.661296:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484291456 00000100:00100000:2.0:1789760763.661352:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752901072507 time 752949422245 state 0x0 ref 1 00000100:00100000:2.0:1789760763.661368:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752949682661 time 752949422245 state 0x1 ref 1 00000100:00100000:2.0:1789760763.661450:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752949682661 time 752992379389 state 0x0 ref 1 00000100:00100000:2.0:1789760763.661463:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b82300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484291456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760763.661510:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23150592, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:2.0:1789760763.661518:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.661534:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=752s v=(0 0) 00010000:00010000:2.0:1789760763.661539:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760763.661544:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23150592->23154687] (req 23150592->23154687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.661557:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23150592->23154687] (req 23150592->23154687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760763.691232:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972970 is committed 00002000:00010000:1.0:1789760763.691286:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23150592 -> 23154688 00010000:00010000:1.0:1789760763.691303:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23150592->23154687] (req 23150592->23154687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.691315:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23150592->23154687] (req 23150592->23154687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.691329:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23150592->23154687] (req 23150592->23154687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.691335:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760763.691340:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23150592->23154687] (req 23150592->23154687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.691348:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23150592->23154687] (req 23150592->23154687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.691369:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23150592->23154687] (req 23150592->23154687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.691376:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209449 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23150592->23154687] (req 23150592->23154687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760763.691704:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b82300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484291456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30237us (30560us total) trans 4294972970 rc 0/0 00000100:00100000:1.0:1789760763.691715:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5681 00000100:00100000:1.0:1789760763.691725:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752949682661 time 752992379389 state 0x2 ref 0 00000100:00100000:1.0:1789760763.698651:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.698967:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484291584 00000100:00100000:3.0:1789760763.698977:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484291584 00000100:00100000:3.0:1789760763.699033:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752949682661 time 752992379389 state 0x0 ref 1 00000100:00100000:3.0:1789760763.699049:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752992639805 time 752992379389 state 0x1 ref 1 00000100:00100000:3.0:1789760763.699179:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752992639805 time 753030108738 state 0x0 ref 1 00000100:00100000:3.0:1789760763.699192:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484291584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760763.699243:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23154688, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:3.0:1789760763.699252:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.699268:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:3.0:1789760763.699273:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760763.699279:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23154688->23158783] (req 23154688->23158783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.699292:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23154688->23158783] (req 23154688->23158783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760763.743302:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972971 is committed 00002000:00010000:1.0:1789760763.743491:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23154688 -> 23158784 00010000:00010000:1.0:1789760763.743509:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23154688->23158783] (req 23154688->23158783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.743528:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23154688->23158783] (req 23154688->23158783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.743537:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23154688->23158783] (req 23154688->23158783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.743544:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760763.743549:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23154688->23158783] (req 23154688->23158783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.743559:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23154688->23158783] (req 23154688->23158783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.743580:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23154688->23158783] (req 23154688->23158783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760763.743589:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209450 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23154688->23158783] (req 23154688->23158783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760763.744383:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484291584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45185us (45733us total) trans 4294972971 rc 0/0 00000100:00100000:1.0:1789760763.744395:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5682 00000100:00100000:1.0:1789760763.744406:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752992639805 time 753030108738 state 0x2 ref 0 00000100:00100000:1.0:1789760763.748467:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.748543:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484291712 00000100:00100000:3.0:1789760763.748553:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484291712 00000100:00100000:3.0:1789760763.748606:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 752992639805 time 753030108738 state 0x0 ref 1 00000100:00100000:3.0:1789760763.748622:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753030369154 time 753030108738 state 0x1 ref 1 00000100:00100000:3.0:1789760763.748653:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753030369154 time 753079584430 state 0x0 ref 1 00000100:00100000:3.0:1789760763.748666:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484291712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760763.748726:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23158784, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:3.0:1789760763.748736:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.748751:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:3.0:1789760763.748756:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760763.748762:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23158784->23162879] (req 23158784->23162879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.748776:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23158784->23162879] (req 23158784->23162879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760763.784666:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972972 is committed 00002000:00010000:2.0:1789760763.784778:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23158784 -> 23162880 00010000:00010000:2.0:1789760763.784794:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23158784->23162879] (req 23158784->23162879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.784806:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23158784->23162879] (req 23158784->23162879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.784819:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23158784->23162879] (req 23158784->23162879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.784826:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760763.784831:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23158784->23162879] (req 23158784->23162879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.784839:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23158784->23162879] (req 23158784->23162879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.784860:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23158784->23162879] (req 23158784->23162879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.784867:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209457 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23158784->23162879] (req 23158784->23162879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760763.784975:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484291712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36305us (36512us total) trans 4294972972 rc 0/0 00000100:00100000:2.0:1789760763.784986:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5683 00000100:00100000:2.0:1789760763.784997:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753030369154 time 753079584430 state 0x2 ref 0 00000100:00100000:1.0:1789760763.788537:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.788559:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8ab8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508931456:0@lo:13:osp-pre-0-0.0 00000100:00100000:1.0:1789760763.788642:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760763.788711:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508931456 00000100:00100000:1.0:1789760763.788718:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508931456 00000100:00100000:1.0:1789760763.788768:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 398 00000100:00100000:1.0:1789760763.788778:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b82d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508931456:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:1.0:1789760763.788804:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 526581760 pending 0 free 3834642432 avail 3830448128 00000100:00100000:1.0:1789760763.788850:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b82d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508931456:12345-0@lo:13:osp-pre-0-0.0 Request processed in 73us (207us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760763.788858:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 398 00000100:00100000:1.0:1789760763.788907:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8ab8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508931456:0@lo:13:osp-pre-0-0.0 00010000:00010000:3.0:1789760763.789721:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760763.791223:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfbb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508931584:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760763.791301:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760763.791361:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1789760763.791367:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5115c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508931712:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760763.791406:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760763.791629:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508931584 00000100:00100000:2.0:1789760763.791638:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508931584 00000100:00100000:2.0:1789760763.791698:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 399 00000100:00100000:2.0:1789760763.791709:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508931584:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:2.0:1789760763.791734:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 526581760 pending 0 free 3834642432 avail 3830448128 00000100:00100000:2.0:1789760763.791786:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508931584:12345-0@lo:13:osp-pre-0-1.0 Request processed in 80us (484us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760763.791792:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 399 00000100:00100000:2.0:1789760763.791836:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfbb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508931584:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760763.793199:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508931712 00000100:00100000:0.0:1789760763.793211:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508931712 00000100:00100000:0.0:1789760763.793339:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 457 00000100:00100000:0.0:1789760763.793350:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508931712:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:0.0:1789760763.793496:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:0.0:1789760763.793576:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508931712:12345-0@lo:41:osp-pre-0-1.0 Request processed in 229us (2169us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760763.793584:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 457 00000100:00100000:2.0:1789760763.793671:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5115c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508931712:0@lo:41:osp-pre-0-1.0 00000100:00100000:2.0:1789760763.794453:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760763.794626:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484291840 00000100:00100000:3.0:1789760763.794634:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484291840 00000100:00100000:3.0:1789760763.794705:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753030369154 time 753079584430 state 0x0 ref 1 00000100:00100000:3.0:1789760763.794720:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753079844846 time 753079584430 state 0x1 ref 1 00000100:00100000:3.0:1789760763.794749:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753079844846 time 753125679425 state 0x0 ref 1 00000100:00100000:3.0:1789760763.794761:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484291840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760763.794811:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23162880, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:3.0:1789760763.794820:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.794837:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:3.0:1789760763.794842:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760763.794849:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23162880->23166975] (req 23162880->23166975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.794863:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23162880->23166975] (req 23162880->23166975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760763.861389:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23162880 -> 23166976 00000001:00080000:1.0:1789760763.861397:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972973 is committed 00010000:00010000:3.0:1789760763.861410:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23162880->23166975] (req 23162880->23166975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.861423:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23162880->23166975] (req 23162880->23166975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.861431:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23162880->23166975] (req 23162880->23166975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.861438:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760763.861442:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23162880->23166975] (req 23162880->23166975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.861450:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23162880->23166975] (req 23162880->23166975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.861468:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23162880->23166975] (req 23162880->23166975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760763.861475:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620945e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23162880->23166975] (req 23162880->23166975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760763.861569:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484291840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66805us (67118us total) trans 4294972973 rc 0/0 00000100:00100000:3.0:1789760763.861583:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5684 00000100:00100000:3.0:1789760763.861593:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753079844846 time 753125679425 state 0x2 ref 0 00000100:00100000:1.0:1789760763.866439:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760763.866604:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484291968 00000100:00100000:2.0:1789760763.866614:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484291968 00000100:00100000:2.0:1789760763.866670:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753079844846 time 753125679425 state 0x0 ref 1 00000100:00100000:2.0:1789760763.866685:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753125939841 time 753125679425 state 0x1 ref 1 00000100:00100000:2.0:1789760763.866716:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753125939841 time 753197647095 state 0x0 ref 1 00000100:00100000:2.0:1789760763.866728:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484291968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760763.866773:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23166976, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:2.0:1789760763.866782:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.866797:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:2.0:1789760763.866801:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760763.866807:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23166976->23171071] (req 23166976->23171071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.866820:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23166976->23171071] (req 23166976->23171071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760763.916884:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972974 is committed 00002000:00010000:2.0:1789760763.917242:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23166976 -> 23171072 00010000:00010000:2.0:1789760763.917266:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23166976->23171071] (req 23166976->23171071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.917279:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23166976->23171071] (req 23166976->23171071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.917289:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23166976->23171071] (req 23166976->23171071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.917296:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760763.917301:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23166976->23171071] (req 23166976->23171071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.917310:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23166976->23171071] (req 23166976->23171071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.917331:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23166976->23171071] (req 23166976->23171071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.917339:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209465 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23166976->23171071] (req 23166976->23171071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760763.917437:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484291968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50706us (51003us total) trans 4294972974 rc 0/0 00000100:00100000:2.0:1789760763.917451:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5685 00000100:00100000:2.0:1789760763.917461:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753125939841 time 753197647095 state 0x2 ref 0 00000100:00100000:1.0:1789760763.930492:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760763.930951:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484292096 00000100:00100000:2.0:1789760763.930963:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484292096 00000100:00100000:2.0:1789760763.931020:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753125939841 time 753197647095 state 0x0 ref 1 00000100:00100000:2.0:1789760763.931036:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753197907511 time 753197647095 state 0x1 ref 1 00000100:00100000:2.0:1789760763.931113:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753197907511 time 753262043399 state 0x0 ref 1 00000100:00100000:2.0:1789760763.931128:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484292096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760763.931187:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23171072, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:2.0:1789760763.931197:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.931222:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:2.0:1789760763.931227:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760763.931232:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23171072->23175167] (req 23171072->23175167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.931252:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23171072->23175167] (req 23171072->23175167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760763.977777:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972975 is committed 00002000:00010000:2.0:1789760763.977882:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23171072 -> 23175168 00010000:00010000:2.0:1789760763.977898:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23171072->23175167] (req 23171072->23175167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.977912:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23171072->23175167] (req 23171072->23175167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.977921:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23171072->23175167] (req 23171072->23175167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.978001:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760763.978008:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23171072->23175167] (req 23171072->23175167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.978018:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23171072->23175167] (req 23171072->23175167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.978038:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23171072->23175167] (req 23171072->23175167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760763.978046:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620946c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23171072->23175167] (req 23171072->23175167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760763.978230:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484292096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47100us (47743us total) trans 4294972975 rc 0/0 00000100:00100000:2.0:1789760763.978241:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5686 00000100:00100000:2.0:1789760763.978259:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753197907511 time 753262043399 state 0x2 ref 0 00000100:00100000:1.0:1789760763.984853:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760763.985860:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484292224 00000100:00100000:0.0:1789760763.985873:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484292224 00000100:00100000:0.0:1789760763.985938:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753197907511 time 753262043399 state 0x0 ref 1 00000100:00100000:0.0:1789760763.985953:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753262303815 time 753262043399 state 0x1 ref 1 00000100:00100000:0.0:1789760763.985988:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753262303815 time 753316918800 state 0x0 ref 1 00000100:00100000:0.0:1789760763.986002:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484292224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760763.986051:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23175168, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:0.0:1789760763.986061:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760763.986107:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:0.0:1789760763.986112:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760763.986118:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23175168->23179263] (req 23175168->23179263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760763.986132:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23175168->23179263] (req 23175168->23179263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760764.020557:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972976 is committed 00002000:00010000:2.0:1789760764.021595:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23175168 -> 23179264 00010000:00010000:2.0:1789760764.021612:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23175168->23179263] (req 23175168->23179263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.021626:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23175168->23179263] (req 23175168->23179263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.021636:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23175168->23179263] (req 23175168->23179263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.021644:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760764.021649:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23175168->23179263] (req 23175168->23179263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.021658:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23175168->23179263] (req 23175168->23179263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.021678:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23175168->23179263] (req 23175168->23179263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.021686:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209473 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23175168->23179263] (req 23175168->23179263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760764.021782:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484292224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35779us (36934us total) trans 4294972976 rc 0/0 00000100:00100000:2.0:1789760764.021794:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5687 00000100:00100000:2.0:1789760764.021803:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753262303815 time 753316918800 state 0x2 ref 0 00000100:00100000:3.0:1789760764.026306:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760764.026441:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484292352 00000100:00100000:2.0:1789760764.026452:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484292352 00000100:00100000:2.0:1789760764.026524:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753262303815 time 753316918800 state 0x0 ref 1 00000100:00100000:2.0:1789760764.026540:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753317179216 time 753316918800 state 0x1 ref 1 00000100:00100000:2.0:1789760764.026564:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753317179216 time 753357495081 state 0x0 ref 1 00000100:00100000:2.0:1789760764.026576:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484292352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760764.026624:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23179264, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:2.0:1789760764.026633:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.026649:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:2.0:1789760764.026654:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760764.026660:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23179264->23183359] (req 23179264->23183359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.026672:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23179264->23183359] (req 23179264->23183359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760764.027164:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:0.0:1789760764.077630:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972977 is committed 00002000:00010000:2.0:1789760764.077741:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23179264 -> 23183360 00010000:00010000:2.0:1789760764.077757:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23179264->23183359] (req 23179264->23183359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.077769:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23179264->23183359] (req 23179264->23183359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.077779:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23179264->23183359] (req 23179264->23183359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.077788:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760764.077793:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23179264->23183359] (req 23179264->23183359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.077804:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23179264->23183359] (req 23179264->23183359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.077840:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23179264->23183359] (req 23179264->23183359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.077848:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620947a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23179264->23183359] (req 23179264->23183359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760764.077954:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484292352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51375us (51654us total) trans 4294972977 rc 0/0 00000100:00100000:2.0:1789760764.077965:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5688 00000100:00100000:2.0:1789760764.077975:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753317179216 time 753357495081 state 0x2 ref 0 00000100:00100000:2.0:1789760764.081609:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760764.081686:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484292480 00000100:00100000:1.0:1789760764.081698:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484292480 00000100:00100000:1.0:1789760764.081755:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753317179216 time 753357495081 state 0x0 ref 1 00000100:00100000:1.0:1789760764.081770:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753357755497 time 753357495081 state 0x1 ref 1 00000100:00100000:1.0:1789760764.081801:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753357755497 time 753412731437 state 0x0 ref 1 00000100:00100000:1.0:1789760764.081813:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484292480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760764.081879:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23183360, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:1.0:1789760764.081888:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.081904:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:1.0:1789760764.081908:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760764.081914:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23183360->23187455] (req 23183360->23187455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.081938:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23183360->23187455] (req 23183360->23187455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760764.131642:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972978 is committed 00002000:00010000:1.0:1789760764.131729:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23183360 -> 23187456 00010000:00010000:1.0:1789760764.131744:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23183360->23187455] (req 23183360->23187455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.131758:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23183360->23187455] (req 23183360->23187455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.131767:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23183360->23187455] (req 23183360->23187455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.131775:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.131780:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23183360->23187455] (req 23183360->23187455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.131790:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23183360->23187455] (req 23183360->23187455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.132203:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23183360->23187455] (req 23183360->23187455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.132215:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209481 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23183360->23187455] (req 23183360->23187455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.132414:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484292480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50598us (50811us total) trans 4294972978 rc 0/0 00000100:00100000:1.0:1789760764.132424:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5689 00000100:00100000:1.0:1789760764.132433:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753357755497 time 753412731437 state 0x2 ref 0 00000100:00100000:0.0:1789760764.135579:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760764.135695:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484292608 00000100:00100000:2.0:1789760764.135704:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484292608 00000100:00100000:2.0:1789760764.135755:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753357755497 time 753412731437 state 0x0 ref 1 00000100:00100000:2.0:1789760764.135772:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753412991853 time 753412731437 state 0x1 ref 1 00000100:00100000:2.0:1789760764.135884:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753412991853 time 753466813653 state 0x0 ref 1 00000100:00100000:2.0:1789760764.135898:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484292608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760764.135955:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23187456, mtime: 0x6aad94fb, atime: 0x0, ctime: 0x6aad94fb, blocks: 0xb807 00010000:00010000:2.0:1789760764.135964:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.135980:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:2.0:1789760764.135984:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760764.135991:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23187456->23191551] (req 23187456->23191551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.136004:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23187456->23191551] (req 23187456->23191551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760764.160115:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972979 is committed 00002000:00010000:2.0:1789760764.160197:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23187456 -> 23191552 00002000:00010000:2.0:1789760764.160201:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760763 -> 1789760764 00002000:00010000:2.0:1789760764.160204:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760763 -> 1789760764 00010000:00010000:2.0:1789760764.160217:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23187456->23191551] (req 23187456->23191551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.160231:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23187456->23191551] (req 23187456->23191551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.160324:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23187456->23191551] (req 23187456->23191551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.160333:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760764.160338:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23187456->23191551] (req 23187456->23191551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.160348:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23187456->23191551] (req 23187456->23191551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.160368:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23187456->23191551] (req 23187456->23191551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.160377:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209488 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23187456->23191551] (req 23187456->23191551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760764.160514:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484292608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24613us (24938us total) trans 4294972979 rc 0/0 00000100:00100000:2.0:1789760764.160525:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5690 00000100:00100000:2.0:1789760764.160535:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753412991853 time 753466813653 state 0x2 ref 0 00000100:00100000:0.0:1789760764.164326:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760764.164401:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484292736 00000100:00100000:2.0:1789760764.164411:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484292736 00000100:00100000:2.0:1789760764.164461:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753412991853 time 753466813653 state 0x0 ref 1 00000100:00100000:2.0:1789760764.164476:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753467074069 time 753466813653 state 0x1 ref 1 00000100:00100000:2.0:1789760764.164497:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753467074069 time 753495428734 state 0x0 ref 1 00000100:00100000:2.0:1789760764.164508:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484292736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760764.164555:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23191552, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760764.164564:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.164580:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:2.0:1789760764.164584:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760764.164591:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23191552->23195647] (req 23191552->23195647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.164605:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23191552->23195647] (req 23191552->23195647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760764.204797:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972980 is committed 00002000:00010000:0.0:1789760764.205248:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23191552 -> 23195648 00010000:00010000:0.0:1789760764.205266:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23191552->23195647] (req 23191552->23195647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.205279:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23191552->23195647] (req 23191552->23195647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.205288:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23191552->23195647] (req 23191552->23195647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.205296:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760764.205301:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23191552->23195647] (req 23191552->23195647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.205310:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23191552->23195647] (req 23191552->23195647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.205332:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23191552->23195647] (req 23191552->23195647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.205349:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620948f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23191552->23195647] (req 23191552->23195647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760764.205454:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484292736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40942us (41132us total) trans 4294972980 rc 0/0 00000100:00100000:0.0:1789760764.205466:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5691 00000100:00100000:0.0:1789760764.205475:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753467074069 time 753495428734 state 0x2 ref 0 00000100:00100000:1.0:1789760764.208642:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760764.208823:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484292864 00000100:00100000:0.0:1789760764.208833:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484292864 00000100:00100000:0.0:1789760764.208889:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753467074069 time 753495428734 state 0x0 ref 1 00000100:00100000:0.0:1789760764.208905:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753495689150 time 753495428734 state 0x1 ref 1 00000100:00100000:0.0:1789760764.208940:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753495689150 time 753539871038 state 0x0 ref 1 00000100:00100000:0.0:1789760764.208952:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484292864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760764.208997:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23195648, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:0.0:1789760764.209007:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.209023:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:0.0:1789760764.209028:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760764.209033:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23195648->23199743] (req 23195648->23199743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.209195:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23195648->23199743] (req 23195648->23199743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760764.265986:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972981 is committed 00002000:00010000:0.0:1789760764.266107:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23195648 -> 23199744 00010000:00010000:0.0:1789760764.266123:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23195648->23199743] (req 23195648->23199743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.266136:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23195648->23199743] (req 23195648->23199743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.266153:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23195648->23199743] (req 23195648->23199743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.266161:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760764.266166:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23195648->23199743] (req 23195648->23199743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.266175:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23195648->23199743] (req 23195648->23199743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.266194:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23195648->23199743] (req 23195648->23199743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.266203:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209496 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23195648->23199743] (req 23195648->23199743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760764.266305:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484292864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57349us (57668us total) trans 4294972981 rc 0/0 00000100:00100000:0.0:1789760764.266315:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5692 00000100:00100000:0.0:1789760764.266325:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753495689150 time 753539871038 state 0x2 ref 0 00000100:00100000:1.0:1789760764.270001:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760764.270182:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484292992 00000100:00100000:2.0:1789760764.270192:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484292992 00000100:00100000:2.0:1789760764.270313:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753495689150 time 753539871038 state 0x0 ref 1 00000100:00100000:2.0:1789760764.270328:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753540131454 time 753539871038 state 0x1 ref 1 00000100:00100000:2.0:1789760764.270386:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753540131454 time 753601316306 state 0x0 ref 1 00000100:00100000:2.0:1789760764.270400:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a32d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484292992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760764.270446:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23199744, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760764.270454:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.270469:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:2.0:1789760764.270473:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760764.270479:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23199744->23203839] (req 23199744->23203839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.270493:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23199744->23203839] (req 23199744->23203839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.300464:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760764.300484:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508931840:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760764.300547:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760764.300614:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508931840 00000100:00100000:1.0:1789760764.300622:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508931840 00000100:00100000:1.0:1789760764.300666:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 458 00000100:00100000:1.0:1789760764.300675:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a33800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508931840:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760764.300711:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760764.300757:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a33800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508931840:12345-0@lo:41:osp-pre-1-0.0 Request processed in 82us (210us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760764.300765:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 458 00000100:00100000:1.0:1789760764.300812:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508931840:0@lo:41:osp-pre-1-0.0 00000001:00080000:1.0:1789760764.310240:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972982 is committed 00002000:00010000:0.0:1789760764.310933:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23199744 -> 23203840 00010000:00010000:0.0:1789760764.310951:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23199744->23203839] (req 23199744->23203839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.310968:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23199744->23203839] (req 23199744->23203839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.310977:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23199744->23203839] (req 23199744->23203839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.310984:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760764.310989:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23199744->23203839] (req 23199744->23203839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.310997:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23199744->23203839] (req 23199744->23203839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.311017:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23199744->23203839] (req 23199744->23203839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.311025:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620949d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23199744->23203839] (req 23199744->23203839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760764.311750:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a32d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484292992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41345us (41750us total) trans 4294972982 rc 0/0 00000100:00100000:0.0:1789760764.311761:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5693 00000100:00100000:0.0:1789760764.311783:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753540131454 time 753601316306 state 0x2 ref 0 00000100:00100000:1.0:1789760764.316634:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760764.316779:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484293120 00000100:00100000:2.0:1789760764.316788:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484293120 00000100:00100000:2.0:1789760764.316840:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753540131454 time 753601316306 state 0x0 ref 1 00000100:00100000:2.0:1789760764.316856:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753601576722 time 753601316306 state 0x1 ref 1 00000100:00100000:2.0:1789760764.316932:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753601576722 time 753647861425 state 0x0 ref 1 00000100:00100000:2.0:1789760764.316946:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484293120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760764.316995:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23203840, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760764.317004:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.317020:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:2.0:1789760764.317025:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760764.317031:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23203840->23207935] (req 23203840->23207935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.317046:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23203840->23207935] (req 23203840->23207935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760764.359289:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972983 is committed 00002000:00010000:1.0:1789760764.359512:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23203840 -> 23207936 00010000:00010000:1.0:1789760764.359528:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23203840->23207935] (req 23203840->23207935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.359542:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23203840->23207935] (req 23203840->23207935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.359550:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23203840->23207935] (req 23203840->23207935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.359558:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.359563:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23203840->23207935] (req 23203840->23207935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.359572:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23203840->23207935] (req 23203840->23207935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.359593:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23203840->23207935] (req 23203840->23207935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.359601:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962094a4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23203840->23207935] (req 23203840->23207935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.359828:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484293120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42880us (43198us total) trans 4294972983 rc 0/0 00000100:00100000:1.0:1789760764.359842:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5694 00000100:00100000:1.0:1789760764.359853:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753601576722 time 753647861425 state 0x2 ref 0 00000100:00100000:2.0:1789760764.366577:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760764.366767:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484293248 00000100:00100000:0.0:1789760764.366778:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484293248 00000100:00100000:0.0:1789760764.366835:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753601576722 time 753647861425 state 0x0 ref 1 00000100:00100000:0.0:1789760764.366849:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753648121841 time 753647861425 state 0x1 ref 1 00000100:00100000:0.0:1789760764.366912:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753648121841 time 753697841548 state 0x0 ref 1 00000100:00100000:0.0:1789760764.366990:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484293248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760764.367124:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23207936, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:0.0:1789760764.367134:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.367160:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:0.0:1789760764.367165:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760764.367172:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23207936->23212031] (req 23207936->23212031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.367186:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23207936->23212031] (req 23207936->23212031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760764.427618:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972984 is committed 00002000:00010000:1.0:1789760764.427836:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23207936 -> 23212032 00010000:00010000:1.0:1789760764.427850:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23207936->23212031] (req 23207936->23212031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.427863:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23207936->23212031] (req 23207936->23212031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.427872:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23207936->23212031] (req 23207936->23212031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.427878:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.427883:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23207936->23212031] (req 23207936->23212031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.427892:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23207936->23212031] (req 23207936->23212031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.427910:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23207936->23212031] (req 23207936->23212031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.427918:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962094ab lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23207936->23212031] (req 23207936->23212031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.428105:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484293248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61177us (61532us total) trans 4294972984 rc 0/0 00000100:00100000:1.0:1789760764.428116:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5695 00000100:00100000:1.0:1789760764.428126:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753648121841 time 753697841548 state 0x2 ref 0 00000100:00100000:2.0:1789760764.442780:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760764.442876:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484293376 00000100:00100000:0.0:1789760764.442886:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484293376 00000100:00100000:0.0:1789760764.442960:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753648121841 time 753697841548 state 0x0 ref 1 00000100:00100000:0.0:1789760764.442976:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753698101964 time 753697841548 state 0x1 ref 1 00000100:00100000:0.0:1789760764.443111:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753698101964 time 753773993838 state 0x0 ref 1 00000100:00100000:0.0:1789760764.443130:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484293376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760764.443193:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23212032, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:0.0:1789760764.443203:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.443218:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:0.0:1789760764.443222:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760764.443228:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23212032->23216127] (req 23212032->23216127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.443241:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23212032->23216127] (req 23212032->23216127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760764.482156:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972985 is committed 00002000:00010000:1.0:1789760764.482540:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23212032 -> 23216128 00010000:00010000:1.0:1789760764.482557:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23212032->23216127] (req 23212032->23216127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.482570:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23212032->23216127] (req 23212032->23216127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.482579:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23212032->23216127] (req 23212032->23216127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.482586:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.482591:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23212032->23216127] (req 23212032->23216127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.482600:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23212032->23216127] (req 23212032->23216127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.482620:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23212032->23216127] (req 23212032->23216127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.482633:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962094b2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23212032->23216127] (req 23212032->23216127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.482884:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484293376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39754us (40109us total) trans 4294972985 rc 0/0 00000100:00100000:1.0:1789760764.482895:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5696 00000100:00100000:1.0:1789760764.482905:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753698101964 time 753773993838 state 0x2 ref 0 00000100:00100000:0.0:1789760764.487295:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760764.487384:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484293504 00000100:00100000:2.0:1789760764.487394:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484293504 00000100:00100000:2.0:1789760764.487446:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753698101964 time 753773993838 state 0x0 ref 1 00000100:00100000:2.0:1789760764.487461:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753774254254 time 753773993838 state 0x1 ref 1 00000100:00100000:2.0:1789760764.487533:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753774254254 time 753818463263 state 0x0 ref 1 00000100:00100000:2.0:1789760764.487546:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484293504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760764.487596:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23216128, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760764.487605:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.487621:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:2.0:1789760764.487625:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760764.487631:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23216128->23220223] (req 23216128->23220223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.487645:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23216128->23220223] (req 23216128->23220223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760764.540482:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972986 is committed 00002000:00010000:2.0:1789760764.540615:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23216128 -> 23220224 00010000:00010000:2.0:1789760764.540630:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23216128->23220223] (req 23216128->23220223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.540643:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23216128->23220223] (req 23216128->23220223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.540653:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23216128->23220223] (req 23216128->23220223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.540736:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760764.540743:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23216128->23220223] (req 23216128->23220223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.540753:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23216128->23220223] (req 23216128->23220223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.540776:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23216128->23220223] (req 23216128->23220223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.540847:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf38263962094b9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23216128->23220223] (req 23216128->23220223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760764.541134:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484293504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53486us (53805us total) trans 4294972986 rc 0/0 00000100:00100000:2.0:1789760764.541158:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5697 00000100:00100000:2.0:1789760764.541169:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753774254254 time 753818463263 state 0x2 ref 0 00000100:00100000:2.0:1789760764.545894:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760764.546505:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484293632 00000100:00100000:1.0:1789760764.546516:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484293632 00000100:00100000:1.0:1789760764.546573:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753774254254 time 753818463263 state 0x0 ref 1 00000100:00100000:1.0:1789760764.546590:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753818723679 time 753818463263 state 0x1 ref 1 00000100:00100000:1.0:1789760764.546651:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753818723679 time 753877581249 state 0x0 ref 1 00000100:00100000:1.0:1789760764.546740:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484293632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760764.547322:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23220224, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:1.0:1789760764.547336:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.547353:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:1.0:1789760764.547358:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760764.547365:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23220224->23224319] (req 23220224->23224319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.547383:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23220224->23224319] (req 23220224->23224319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760764.586701:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972987 is committed 00002000:00010000:1.0:1789760764.586853:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23220224 -> 23224320 00010000:00010000:1.0:1789760764.586869:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23220224->23224319] (req 23220224->23224319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.586882:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23220224->23224319] (req 23220224->23224319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.586891:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23220224->23224319] (req 23220224->23224319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.586898:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.586903:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23220224->23224319] (req 23220224->23224319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.587211:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23220224->23224319] (req 23220224->23224319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.587231:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23220224->23224319] (req 23220224->23224319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.587239:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962094c0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23220224->23224319] (req 23220224->23224319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.587380:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484293632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40640us (41490us total) trans 4294972987 rc 0/0 00000100:00100000:1.0:1789760764.587392:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5698 00000100:00100000:1.0:1789760764.587403:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753818723679 time 753877581249 state 0x2 ref 0 00000100:00100000:1.0:1789760764.592894:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760764.592987:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484293760 00000100:00100000:0.0:1789760764.592996:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484293760 00000100:00100000:0.0:1789760764.593054:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753818723679 time 753877581249 state 0x0 ref 1 00000100:00100000:0.0:1789760764.593112:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753877841665 time 753877581249 state 0x1 ref 1 00000100:00100000:0.0:1789760764.593135:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753877841665 time 753924066801 state 0x0 ref 1 00000100:00100000:0.0:1789760764.593159:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a32680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484293760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760764.593210:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23224320, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:0.0:1789760764.593219:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.593234:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:0.0:1789760764.593239:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760764.593245:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23224320->23228415] (req 23224320->23228415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.593259:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23224320->23228415] (req 23224320->23228415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760764.634359:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972988 is committed 00002000:00010000:1.0:1789760764.634539:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23224320 -> 23228416 00010000:00010000:1.0:1789760764.634554:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23224320->23228415] (req 23224320->23228415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.634566:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23224320->23228415] (req 23224320->23228415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.634576:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23224320->23228415] (req 23224320->23228415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.634584:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.634588:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23224320->23228415] (req 23224320->23228415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.634597:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23224320->23228415] (req 23224320->23228415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.634616:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23224320->23228415] (req 23224320->23228415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.634630:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf38263962094c7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23224320->23228415] (req 23224320->23228415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.634753:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a32680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484293760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41595us (41866us total) trans 4294972988 rc 0/0 00000100:00100000:1.0:1789760764.634761:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5699 00000100:00100000:1.0:1789760764.634768:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753877841665 time 753924066801 state 0x2 ref 0 00000100:00100000:0.0:1789760764.637539:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760764.637630:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484293888 00000100:00100000:2.0:1789760764.637640:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484293888 00000100:00100000:2.0:1789760764.637693:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753877841665 time 753924066801 state 0x0 ref 1 00000100:00100000:2.0:1789760764.637708:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753924327217 time 753924066801 state 0x1 ref 1 00000100:00100000:2.0:1789760764.637737:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753924327217 time 753968668033 state 0x0 ref 1 00000100:00100000:2.0:1789760764.637748:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484293888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760764.637798:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23228416, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760764.637806:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.637822:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=753s v=(0 0) 00010000:00010000:2.0:1789760764.637828:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760764.637835:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23228416->23232511] (req 23228416->23232511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.637850:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23228416->23232511] (req 23228416->23232511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760764.665526:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972989 is committed 00002000:00010000:1.0:1789760764.665634:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23228416 -> 23232512 00010000:00010000:1.0:1789760764.665650:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23228416->23232511] (req 23228416->23232511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.665664:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23228416->23232511] (req 23228416->23232511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.665673:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23228416->23232511] (req 23228416->23232511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.665682:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.665686:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23228416->23232511] (req 23228416->23232511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.665696:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23228416->23232511] (req 23228416->23232511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.665722:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23228416->23232511] (req 23228416->23232511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.665730:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962094ce lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23228416->23232511] (req 23228416->23232511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.665824:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484293888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28073us (28291us total) trans 4294972989 rc 0/0 00000100:00100000:1.0:1789760764.665835:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5700 00000100:00100000:1.0:1789760764.665844:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753924327217 time 753968668033 state 0x2 ref 0 00000100:00100000:2.0:1789760764.670377:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760764.670464:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484294016 00000100:00100000:3.0:1789760764.670474:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484294016 00000100:00100000:3.0:1789760764.670527:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753924327217 time 753968668033 state 0x0 ref 1 00000100:00100000:3.0:1789760764.670544:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753968928449 time 753968668033 state 0x1 ref 1 00000100:00100000:3.0:1789760764.670575:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753968928449 time 754001506053 state 0x0 ref 1 00000100:00100000:3.0:1789760764.670589:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760764.670637:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23232512, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:3.0:1789760764.670645:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.670660:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:3.0:1789760764.670665:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760764.670671:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23232512->23236607] (req 23232512->23236607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.670687:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23232512->23236607] (req 23232512->23236607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760764.712322:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972990 is committed 00002000:00010000:0.0:1789760764.712424:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23232512 -> 23236608 00010000:00010000:0.0:1789760764.712441:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23232512->23236607] (req 23232512->23236607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.712455:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23232512->23236607] (req 23232512->23236607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.712464:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23232512->23236607] (req 23232512->23236607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.712472:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760764.712477:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23232512->23236607] (req 23232512->23236607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.712486:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23232512->23236607] (req 23232512->23236607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.712506:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23232512->23236607] (req 23232512->23236607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.712514:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962094d5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23232512->23236607] (req 23232512->23236607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760764.712639:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42048us (42267us total) trans 4294972990 rc 0/0 00000100:00100000:0.0:1789760764.712650:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5701 00000100:00100000:0.0:1789760764.712659:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753968928449 time 754001506053 state 0x2 ref 0 00000100:00100000:2.0:1789760764.715814:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760764.716024:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484294144 00000100:00100000:1.0:1789760764.716035:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484294144 00000100:00100000:1.0:1789760764.716119:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 753968928449 time 754001506053 state 0x0 ref 1 00000100:00100000:1.0:1789760764.716134:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754001766469 time 754001506053 state 0x1 ref 1 00000100:00100000:1.0:1789760764.716259:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754001766469 time 754047188766 state 0x0 ref 1 00000100:00100000:1.0:1789760764.716273:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760764.716322:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23236608, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:1.0:1789760764.716330:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.716352:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:1.0:1789760764.716356:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760764.716363:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23236608->23240703] (req 23236608->23240703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.716376:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23236608->23240703] (req 23236608->23240703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760764.746893:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972991 is committed 00002000:00010000:1.0:1789760764.747062:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23236608 -> 23240704 00010000:00010000:1.0:1789760764.747096:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23236608->23240703] (req 23236608->23240703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.747109:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23236608->23240703] (req 23236608->23240703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.747118:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23236608->23240703] (req 23236608->23240703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.747126:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.747130:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23236608->23240703] (req 23236608->23240703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.747139:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23236608->23240703] (req 23236608->23240703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.747169:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23236608->23240703] (req 23236608->23240703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.747177:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094dc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23236608->23240703] (req 23236608->23240703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.747375:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31099us (31563us total) trans 4294972991 rc 0/0 00000100:00100000:1.0:1789760764.747386:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5702 00000100:00100000:1.0:1789760764.747398:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754001766469 time 754047188766 state 0x2 ref 0 00000100:00100000:1.0:1789760764.751572:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760764.751673:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484294272 00000100:00100000:0.0:1789760764.751683:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484294272 00000100:00100000:0.0:1789760764.751746:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754001766469 time 754047188766 state 0x0 ref 1 00000100:00100000:0.0:1789760764.751761:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754047449182 time 754047188766 state 0x1 ref 1 00000100:00100000:0.0:1789760764.751830:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754047449182 time 754082759969 state 0x0 ref 1 00000100:00100000:0.0:1789760764.751845:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760764.751895:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23240704, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:0.0:1789760764.751904:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.751994:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:0.0:1789760764.752000:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760764.752007:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23240704->23244799] (req 23240704->23244799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.752023:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23240704->23244799] (req 23240704->23244799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760764.786818:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972992 is committed 00002000:00010000:0.0:1789760764.786933:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23240704 -> 23244800 00010000:00010000:0.0:1789760764.786950:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23240704->23244799] (req 23240704->23244799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.786964:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23240704->23244799] (req 23240704->23244799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.786973:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23240704->23244799] (req 23240704->23244799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.786980:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760764.786984:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23240704->23244799] (req 23240704->23244799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.786994:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23240704->23244799] (req 23240704->23244799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.787013:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23240704->23244799] (req 23240704->23244799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.787021:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962094e3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23240704->23244799] (req 23240704->23244799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760764.787165:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35319us (35598us total) trans 4294972992 rc 0/0 00000100:00100000:0.0:1789760764.787176:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5703 00000100:00100000:0.0:1789760764.787185:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754047449182 time 754082759969 state 0x2 ref 0 00000100:00100000:1.0:1789760764.790711:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760764.790800:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484294400 00000100:00100000:3.0:1789760764.790810:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484294400 00000100:00100000:3.0:1789760764.790858:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754047449182 time 754082759969 state 0x0 ref 1 00000100:00100000:3.0:1789760764.790873:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754083020385 time 754082759969 state 0x1 ref 1 00000100:00100000:3.0:1789760764.790903:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754083020385 time 754121834133 state 0x0 ref 1 00000100:00100000:3.0:1789760764.790915:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760764.790972:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23244800, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:3.0:1789760764.790981:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.790997:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:3.0:1789760764.791001:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760764.791007:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23244800->23248895] (req 23244800->23248895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.791020:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23244800->23248895] (req 23244800->23248895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.816648:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760764.820013:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972993 is committed 00002000:00010000:1.0:1789760764.820212:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23244800 -> 23248896 00010000:00010000:1.0:1789760764.820232:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23244800->23248895] (req 23244800->23248895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.820245:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23244800->23248895] (req 23244800->23248895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.820253:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23244800->23248895] (req 23244800->23248895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.820261:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.820265:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23244800->23248895] (req 23244800->23248895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.820274:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23244800->23248895] (req 23244800->23248895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.820295:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23244800->23248895] (req 23244800->23248895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.820303:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962094ea lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23244800->23248895] (req 23244800->23248895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.820393:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29476us (29688us total) trans 4294972993 rc 0/0 00000100:00100000:1.0:1789760764.820401:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5704 00000100:00100000:1.0:1789760764.820409:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754083020385 time 754121834133 state 0x2 ref 0 00000100:00100000:3.0:1789760764.827626:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760764.827739:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484294528 00000100:00100000:0.0:1789760764.827749:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484294528 00000100:00100000:0.0:1789760764.827817:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754083020385 time 754121834133 state 0x0 ref 1 00000100:00100000:0.0:1789760764.827832:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754122094549 time 754121834133 state 0x1 ref 1 00000100:00100000:0.0:1789760764.827865:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754122094549 time 754158796318 state 0x0 ref 1 00000100:00100000:0.0:1789760764.827879:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760764.827926:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23248896, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:0.0:1789760764.827936:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.827952:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:0.0:1789760764.827956:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760764.827961:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23248896->23252991] (req 23248896->23252991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.827980:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23248896->23252991] (req 23248896->23252991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760764.857621:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972994 is committed 00002000:00010000:0.0:1789760764.857953:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23248896 -> 23252992 00010000:00010000:0.0:1789760764.857969:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23248896->23252991] (req 23248896->23252991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.857982:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23248896->23252991] (req 23248896->23252991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.857991:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23248896->23252991] (req 23248896->23252991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.857999:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760764.858003:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23248896->23252991] (req 23248896->23252991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.858011:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23248896->23252991] (req 23248896->23252991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.858030:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23248896->23252991] (req 23248896->23252991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760764.858037:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962094f1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23248896->23252991] (req 23248896->23252991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760764.858171:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484294528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30291us (30550us total) trans 4294972994 rc 0/0 00000100:00100000:0.0:1789760764.858182:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5705 00000100:00100000:0.0:1789760764.858191:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754122094549 time 754158796318 state 0x2 ref 0 00000100:00100000:0.0:1789760764.860994:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760764.861116:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484294656 00000100:00100000:1.0:1789760764.861126:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484294656 00000100:00100000:1.0:1789760764.861189:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754122094549 time 754158796318 state 0x0 ref 1 00000100:00100000:1.0:1789760764.861204:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754159056734 time 754158796318 state 0x1 ref 1 00000100:00100000:1.0:1789760764.861234:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754159056734 time 754192164672 state 0x0 ref 1 00000100:00100000:1.0:1789760764.861246:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484294656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760764.861293:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23252992, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:1.0:1789760764.861303:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.861319:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:1.0:1789760764.861323:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760764.861330:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23252992->23257087] (req 23252992->23257087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.861343:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23252992->23257087] (req 23252992->23257087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760764.906606:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972995 is committed 00002000:00010000:1.0:1789760764.906688:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23252992 -> 23257088 00010000:00010000:1.0:1789760764.906705:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23252992->23257087] (req 23252992->23257087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.906718:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23252992->23257087] (req 23252992->23257087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.906725:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23252992->23257087] (req 23252992->23257087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.906732:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760764.906736:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23252992->23257087] (req 23252992->23257087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.906744:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23252992->23257087] (req 23252992->23257087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.906761:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23252992->23257087] (req 23252992->23257087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.906767:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf38263962094f8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23252992->23257087] (req 23252992->23257087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760764.906895:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484294656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45646us (45905us total) trans 4294972995 rc 0/0 00000100:00100000:1.0:1789760764.906906:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5706 00000100:00100000:1.0:1789760764.906916:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754159056734 time 754192164672 state 0x2 ref 0 00000100:00100000:2.0:1789760764.909714:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760764.911007:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484294784 00000100:00100000:0.0:1789760764.911023:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484294784 00000100:00100000:0.0:1789760764.911378:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754159056734 time 754192164672 state 0x0 ref 1 00000100:00100000:0.0:1789760764.911395:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754192425088 time 754192164672 state 0x1 ref 1 00000100:00100000:2.0:1789760764.911711:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754192425088 time 754242639045 state 0x0 ref 1 00000100:00100000:2.0:1789760764.911728:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484294784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760764.911778:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23257088, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760764.911788:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.911804:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:2.0:1789760764.911808:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760764.911813:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23257088->23261183] (req 23257088->23261183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.911827:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23257088->23261183] (req 23257088->23261183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760764.938405:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972996 is committed 00002000:00010000:2.0:1789760764.938510:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23257088 -> 23261184 00010000:00010000:2.0:1789760764.938526:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23257088->23261183] (req 23257088->23261183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.938538:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23257088->23261183] (req 23257088->23261183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.938548:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23257088->23261183] (req 23257088->23261183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.938554:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760764.938558:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23257088->23261183] (req 23257088->23261183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.938566:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23257088->23261183] (req 23257088->23261183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.938586:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23257088->23261183] (req 23257088->23261183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.938594:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962094ff lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23257088->23261183] (req 23257088->23261183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760764.938685:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484294784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26958us (28975us total) trans 4294972996 rc 0/0 00000100:00100000:2.0:1789760764.938695:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5707 00000100:00100000:2.0:1789760764.938705:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754192425088 time 754242639045 state 0x2 ref 0 00000100:00100000:3.0:1789760764.942396:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760764.942652:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484294912 00000100:00100000:1.0:1789760764.942664:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484294912 00000100:00100000:1.0:1789760764.942718:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754192425088 time 754242639045 state 0x0 ref 1 00000100:00100000:1.0:1789760764.942733:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754242899461 time 754242639045 state 0x1 ref 1 00000100:00100000:1.0:1789760764.942772:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754242899461 time 754273702968 state 0x0 ref 1 00000100:00100000:1.0:1789760764.942785:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484294912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760764.942832:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23261184, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:1.0:1789760764.942841:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.942856:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:1.0:1789760764.942861:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760764.942867:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23261184->23265279] (req 23261184->23265279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760764.942880:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23261184->23265279] (req 23261184->23265279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760764.971250:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972997 is committed 00002000:00010000:3.0:1789760764.971372:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23261184 -> 23265280 00010000:00010000:3.0:1789760764.971387:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23261184->23265279] (req 23261184->23265279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.971400:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23261184->23265279] (req 23261184->23265279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.971409:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23261184->23265279] (req 23261184->23265279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.971417:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760764.971421:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23261184->23265279] (req 23261184->23265279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.971430:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23261184->23265279] (req 23261184->23265279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.971450:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23261184->23265279] (req 23261184->23265279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760764.971459:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209506 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23261184->23265279] (req 23261184->23265279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760764.971545:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484294912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28758us (29154us total) trans 4294972997 rc 0/0 00000100:00100000:3.0:1789760764.971555:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5708 00000100:00100000:3.0:1789760764.971565:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754242899461 time 754273702968 state 0x2 ref 0 00000100:00100000:0.0:1789760764.975601:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760764.975698:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484295040 00000100:00100000:2.0:1789760764.975708:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484295040 00000100:00100000:2.0:1789760764.975757:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754242899461 time 754273702968 state 0x0 ref 1 00000100:00100000:2.0:1789760764.975771:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754273963384 time 754273702968 state 0x1 ref 1 00000100:00100000:2.0:1789760764.975802:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754273963384 time 754306733472 state 0x0 ref 1 00000100:00100000:2.0:1789760764.975814:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ad880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484295040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760764.975859:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23265280, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760764.975867:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.975882:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:2.0:1789760764.975886:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760764.975892:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23265280->23269375] (req 23265280->23269375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760764.975904:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23265280->23269375] (req 23265280->23269375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760765.010831:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972998 is committed 00002000:00010000:2.0:1789760765.011012:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23265280 -> 23269376 00010000:00010000:2.0:1789760765.011028:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23265280->23269375] (req 23265280->23269375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.011040:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23265280->23269375] (req 23265280->23269375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.011049:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23265280->23269375] (req 23265280->23269375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.011056:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760765.011062:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23265280->23269375] (req 23265280->23269375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.011101:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23265280->23269375] (req 23265280->23269375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.011122:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23265280->23269375] (req 23265280->23269375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.011130:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620950d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23265280->23269375] (req 23265280->23269375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760765.011225:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ad880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484295040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35408us (35627us total) trans 4294972998 rc 0/0 00000100:00100000:2.0:1789760765.011235:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5709 00000100:00100000:2.0:1789760765.011245:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754273963384 time 754306733472 state 0x2 ref 0 00000100:00100000:0.0:1789760765.015399:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760765.015484:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484295168 00000100:00100000:2.0:1789760765.015493:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484295168 00000100:00100000:2.0:1789760765.015544:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754273963384 time 754306733472 state 0x0 ref 1 00000100:00100000:2.0:1789760765.015558:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754306993888 time 754306733472 state 0x1 ref 1 00000100:00100000:2.0:1789760765.015587:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754306993888 time 754346518113 state 0x0 ref 1 00000100:00100000:2.0:1789760765.015598:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484295168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760765.015643:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23269376, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760765.015652:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.015670:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:2.0:1789760765.015675:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760765.015682:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23269376->23273471] (req 23269376->23273471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.015698:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23269376->23273471] (req 23269376->23273471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760765.043425:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294972999 is committed 00002000:00010000:1.0:1789760765.043510:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23269376 -> 23273472 00010000:00010000:1.0:1789760765.043525:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23269376->23273471] (req 23269376->23273471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.043536:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23269376->23273471] (req 23269376->23273471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.043545:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23269376->23273471] (req 23269376->23273471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.043552:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.043556:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23269376->23273471] (req 23269376->23273471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.043565:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23269376->23273471] (req 23269376->23273471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.043584:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23269376->23273471] (req 23269376->23273471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.043592:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209514 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23269376->23273471] (req 23269376->23273471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.043718:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484295168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28117us (28323us total) trans 4294972999 rc 0/0 00000100:00100000:1.0:1789760765.043733:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5710 00000100:00100000:1.0:1789760765.043742:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754306993888 time 754346518113 state 0x2 ref 0 00000100:00100000:0.0:1789760765.046987:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760765.047195:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484295296 00000100:00100000:2.0:1789760765.047205:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484295296 00000100:00100000:2.0:1789760765.047323:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754306993888 time 754346518113 state 0x0 ref 1 00000100:00100000:2.0:1789760765.047338:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754346778529 time 754346518113 state 0x1 ref 1 00000100:00100000:2.0:1789760765.047407:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754346778529 time 754378337831 state 0x0 ref 1 00000100:00100000:2.0:1789760765.047420:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484295296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760765.047467:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23273472, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760765.047476:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.047492:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:2.0:1789760765.047496:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760765.047502:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23273472->23277567] (req 23273472->23277567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.047516:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23273472->23277567] (req 23273472->23277567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760765.080501:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973000 is committed 00002000:00010000:0.0:1789760765.080622:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23273472 -> 23277568 00010000:00010000:0.0:1789760765.080641:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23273472->23277567] (req 23273472->23277567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.080651:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23273472->23277567] (req 23273472->23277567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.080660:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23273472->23277567] (req 23273472->23277567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.080672:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760765.080677:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23273472->23277567] (req 23273472->23277567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.080685:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23273472->23277567] (req 23273472->23277567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.080704:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23273472->23277567] (req 23273472->23277567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.080710:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620951b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23273472->23277567] (req 23273472->23277567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760765.080839:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12aca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484295296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33417us (33913us total) trans 4294973000 rc 0/0 00000100:00100000:0.0:1789760765.080848:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5711 00000100:00100000:0.0:1789760765.080858:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754346778529 time 754378337831 state 0x2 ref 0 00000100:00100000:0.0:1789760765.085106:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760765.085315:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484295424 00000100:00100000:3.0:1789760765.085325:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484295424 00000100:00100000:3.0:1789760765.085380:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754346778529 time 754378337831 state 0x0 ref 1 00000100:00100000:3.0:1789760765.085396:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754378598247 time 754378337831 state 0x1 ref 1 00000100:00100000:3.0:1789760765.085457:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754378598247 time 754416387047 state 0x0 ref 1 00000100:00100000:3.0:1789760765.085470:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12af800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484295424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760765.085518:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23277568, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:3.0:1789760765.085526:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.085540:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:3.0:1789760765.085544:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760765.085550:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23277568->23281663] (req 23277568->23281663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.085562:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23277568->23281663] (req 23277568->23281663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760765.132545:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23277568 -> 23281664 00010000:00010000:1.0:1789760765.132564:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23277568->23281663] (req 23277568->23281663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.132576:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23277568->23281663] (req 23277568->23281663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.132586:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23277568->23281663] (req 23277568->23281663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.132594:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.132598:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23277568->23281663] (req 23277568->23281663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.132608:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23277568->23281663] (req 23277568->23281663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.132628:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23277568->23281663] (req 23277568->23281663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.132636:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209522 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23277568->23281663] (req 23277568->23281663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.132733:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12af800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484295424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47260us (47667us total) trans 4294973001 rc 0/0 00000100:00100000:1.0:1789760765.132744:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5712 00000100:00100000:1.0:1789760765.132754:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754378598247 time 754416387047 state 0x2 ref 0 00000001:00080000:2.0:1789760765.136045:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973001 is committed 00000100:00100000:0.0:1789760765.136590:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760765.136692:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484295552 00000100:00100000:2.0:1789760765.136701:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484295552 00000100:00100000:2.0:1789760765.136749:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754378598247 time 754416387047 state 0x0 ref 1 00000100:00100000:2.0:1789760765.136765:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754416647463 time 754416387047 state 0x1 ref 1 00000100:00100000:2.0:1789760765.136826:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754416647463 time 754467756529 state 0x0 ref 1 00000100:00100000:2.0:1789760765.136840:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ad180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484295552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760765.136889:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23281664, mtime: 0x6aad94fc, atime: 0x0, ctime: 0x6aad94fc, blocks: 0xb807 00010000:00010000:2.0:1789760765.136897:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.136913:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:2.0:1789760765.136917:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760765.136998:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23281664->23285759] (req 23281664->23285759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.137015:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23281664->23285759] (req 23281664->23285759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760765.167951:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973002 is committed 00002000:00010000:2.0:1789760765.168151:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23281664 -> 23285760 00002000:00010000:2.0:1789760765.168156:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760764 -> 1789760765 00002000:00010000:2.0:1789760765.168159:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760764 -> 1789760765 00010000:00010000:2.0:1789760765.168172:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23281664->23285759] (req 23281664->23285759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.168184:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23281664->23285759] (req 23281664->23285759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.168193:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23281664->23285759] (req 23281664->23285759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.168200:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760765.168206:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23281664->23285759] (req 23281664->23285759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.168215:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23281664->23285759] (req 23281664->23285759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.168316:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23281664->23285759] (req 23281664->23285759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.168326:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209529 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23281664->23285759] (req 23281664->23285759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760765.168418:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ad180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484295552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31576us (31828us total) trans 4294973002 rc 0/0 00000100:00100000:2.0:1789760765.168429:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5713 00000100:00100000:2.0:1789760765.168438:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754416647463 time 754467756529 state 0x2 ref 0 00000100:00100000:0.0:1789760765.172426:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760765.172621:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484295680 00000100:00100000:1.0:1789760765.172631:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484295680 00000100:00100000:1.0:1789760765.172683:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754416647463 time 754467756529 state 0x0 ref 1 00000100:00100000:1.0:1789760765.172697:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754468016945 time 754467756529 state 0x1 ref 1 00000100:00100000:1.0:1789760765.172768:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754468016945 time 754503698209 state 0x0 ref 1 00000100:00100000:1.0:1789760765.172781:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ac380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484295680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760765.172830:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23285760, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:1.0:1789760765.172838:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.172855:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:1.0:1789760765.172859:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760765.172866:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23285760->23289855] (req 23285760->23289855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.172880:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23285760->23289855] (req 23285760->23289855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760765.206375:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23285760 -> 23289856 00010000:00010000:1.0:1789760765.206396:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23285760->23289855] (req 23285760->23289855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.206408:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23285760->23289855] (req 23285760->23289855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.206416:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23285760->23289855] (req 23285760->23289855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.206424:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.206429:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23285760->23289855] (req 23285760->23289855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.206438:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23285760->23289855] (req 23285760->23289855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.206457:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23285760->23289855] (req 23285760->23289855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.206465:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209530 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23285760->23289855] (req 23285760->23289855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.206555:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ac380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484295680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33771us (34134us total) trans 4294973003 rc 0/0 00000100:00100000:1.0:1789760765.206570:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5714 00000100:00100000:1.0:1789760765.206579:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754468016945 time 754503698209 state 0x2 ref 0 00000001:00080000:3.0:1789760765.206661:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973003 is committed 00000100:00100000:0.0:1789760765.209668:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760765.209757:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484295808 00000100:00100000:1.0:1789760765.209767:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484295808 00000100:00100000:1.0:1789760765.209819:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754468016945 time 754503698209 state 0x0 ref 1 00000100:00100000:1.0:1789760765.209834:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754503958625 time 754503698209 state 0x1 ref 1 00000100:00100000:1.0:1789760765.209865:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754503958625 time 754540796145 state 0x0 ref 1 00000100:00100000:1.0:1789760765.209877:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484295808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760765.209924:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23289856, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:1.0:1789760765.209943:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.209958:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:1.0:1789760765.209963:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760765.209969:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23289856->23293951] (req 23289856->23293951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.209983:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23289856->23293951] (req 23289856->23293951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760765.240463:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973004 is committed 00002000:00010000:1.0:1789760765.240621:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23289856 -> 23293952 00010000:00010000:1.0:1789760765.240637:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23289856->23293951] (req 23289856->23293951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.240652:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23289856->23293951] (req 23289856->23293951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.240661:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23289856->23293951] (req 23289856->23293951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.240669:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.240673:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23289856->23293951] (req 23289856->23293951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.240682:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23289856->23293951] (req 23289856->23293951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.240703:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23289856->23293951] (req 23289856->23293951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.240711:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209537 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23289856->23293951] (req 23289856->23293951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.240799:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12adf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484295808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30920us (31137us total) trans 4294973004 rc 0/0 00000100:00100000:1.0:1789760765.240810:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5715 00000100:00100000:1.0:1789760765.240819:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754503958625 time 754540796145 state 0x2 ref 0 00000100:00100000:0.0:1789760765.244993:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760765.245298:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484295936 00000100:00100000:3.0:1789760765.245311:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484295936 00000100:00100000:3.0:1789760765.245362:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754503958625 time 754540796145 state 0x0 ref 1 00000100:00100000:3.0:1789760765.245376:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754541056561 time 754540796145 state 0x1 ref 1 00000100:00100000:3.0:1789760765.245454:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754541056561 time 754576384145 state 0x0 ref 1 00000100:00100000:3.0:1789760765.245467:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484295936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760765.245512:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23293952, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:3.0:1789760765.245520:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.245536:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:3.0:1789760765.245540:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760765.245547:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23293952->23298047] (req 23293952->23298047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.245560:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23293952->23298047] (req 23293952->23298047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760765.300350:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973005 is committed 00002000:00010000:1.0:1789760765.300582:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23293952 -> 23298048 00010000:00010000:1.0:1789760765.300598:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23293952->23298047] (req 23293952->23298047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.300613:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23293952->23298047] (req 23293952->23298047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.300622:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23293952->23298047] (req 23293952->23298047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.300630:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.300636:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23293952->23298047] (req 23293952->23298047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.300646:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23293952->23298047] (req 23293952->23298047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.300665:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23293952->23298047] (req 23293952->23298047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.300674:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620953e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23293952->23298047] (req 23293952->23298047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.300771:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484295936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55302us (55782us total) trans 4294973005 rc 0/0 00000100:00100000:1.0:1789760765.300782:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5716 00000100:00100000:1.0:1789760765.300792:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754541056561 time 754576384145 state 0x2 ref 0 00000100:00100000:0.0:1789760765.304932:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760765.305037:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484296064 00000100:00100000:2.0:1789760765.305049:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484296064 00000100:00100000:2.0:1789760765.305156:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754541056561 time 754576384145 state 0x0 ref 1 00000100:00100000:2.0:1789760765.305172:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754576644561 time 754576384145 state 0x1 ref 1 00000100:00100000:2.0:1789760765.305197:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754576644561 time 754636128286 state 0x0 ref 1 00000100:00100000:2.0:1789760765.305210:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484296064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760765.305259:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23298048, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:2.0:1789760765.305553:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.305570:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:2.0:1789760765.305575:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760765.305580:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23298048->23302143] (req 23298048->23302143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.305595:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23298048->23302143] (req 23298048->23302143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760765.371722:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973006 is committed 00002000:00010000:1.0:1789760765.371876:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23298048 -> 23302144 00010000:00010000:1.0:1789760765.371892:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23298048->23302143] (req 23298048->23302143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.371905:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23298048->23302143] (req 23298048->23302143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.371914:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23298048->23302143] (req 23298048->23302143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.371933:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.371938:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23298048->23302143] (req 23298048->23302143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.371947:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23298048->23302143] (req 23298048->23302143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.371968:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23298048->23302143] (req 23298048->23302143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.371976:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209545 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23298048->23302143] (req 23298048->23302143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.372150:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484296064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66856us (67143us total) trans 4294973006 rc 0/0 00000100:00100000:1.0:1789760765.372162:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5717 00000100:00100000:1.0:1789760765.372172:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754576644561 time 754636128286 state 0x2 ref 0 00000100:00100000:0.0:1789760765.375717:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760765.375867:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484296192 00000100:00100000:2.0:1789760765.375877:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484296192 00000100:00100000:2.0:1789760765.375931:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754576644561 time 754636128286 state 0x0 ref 1 00000100:00100000:2.0:1789760765.375947:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754636388702 time 754636128286 state 0x1 ref 1 00000100:00100000:2.0:1789760765.375977:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754636388702 time 754706908277 state 0x0 ref 1 00000100:00100000:2.0:1789760765.375990:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484296192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760765.376038:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23302144, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:2.0:1789760765.376047:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.376061:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:2.0:1789760765.376101:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760765.376108:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23302144->23306239] (req 23302144->23306239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.376122:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23302144->23306239] (req 23302144->23306239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760765.411275:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973007 is committed 00002000:00010000:2.0:1789760765.411401:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23302144 -> 23306240 00010000:00010000:2.0:1789760765.411417:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23302144->23306239] (req 23302144->23306239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.411436:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23302144->23306239] (req 23302144->23306239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.411446:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23302144->23306239] (req 23302144->23306239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.411454:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760765.411459:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23302144->23306239] (req 23302144->23306239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.411469:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23302144->23306239] (req 23302144->23306239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.411489:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23302144->23306239] (req 23302144->23306239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.411495:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620954c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23302144->23306239] (req 23302144->23306239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760765.411624:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484296192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35631us (35920us total) trans 4294973007 rc 0/0 00000100:00100000:2.0:1789760765.411634:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5718 00000100:00100000:2.0:1789760765.411643:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754636388702 time 754706908277 state 0x2 ref 0 00000100:00100000:0.0:1789760765.417327:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760765.417440:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484296320 00000100:00100000:1.0:1789760765.417449:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484296320 00000100:00100000:1.0:1789760765.417498:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754636388702 time 754706908277 state 0x0 ref 1 00000100:00100000:1.0:1789760765.417512:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754707168693 time 754706908277 state 0x1 ref 1 00000100:00100000:1.0:1789760765.417544:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754707168693 time 754748474411 state 0x0 ref 1 00000100:00100000:1.0:1789760765.417555:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484296320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760765.417602:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23306240, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:1.0:1789760765.417610:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.417623:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:1.0:1789760765.417628:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760765.417633:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23306240->23310335] (req 23306240->23310335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.417645:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23306240->23310335] (req 23306240->23310335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760765.446587:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973008 is committed 00002000:00010000:0.0:1789760765.446680:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23306240 -> 23310336 00010000:00010000:0.0:1789760765.446694:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23306240->23310335] (req 23306240->23310335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.446707:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23306240->23310335] (req 23306240->23310335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.446716:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23306240->23310335] (req 23306240->23310335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.446723:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760765.446728:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23306240->23310335] (req 23306240->23310335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.446736:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23306240->23310335] (req 23306240->23310335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.446756:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23306240->23310335] (req 23306240->23310335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.446765:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209553 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23306240->23310335] (req 23306240->23310335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760765.446856:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484296320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29298us (29533us total) trans 4294973008 rc 0/0 00000100:00100000:0.0:1789760765.446867:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5719 00000100:00100000:0.0:1789760765.446876:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754707168693 time 754748474411 state 0x2 ref 0 00000100:00100000:0.0:1789760765.450008:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760765.450168:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484296448 00000100:00100000:1.0:1789760765.450178:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484296448 00000100:00100000:1.0:1789760765.450289:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754707168693 time 754748474411 state 0x0 ref 1 00000100:00100000:1.0:1789760765.450312:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754748734827 time 754748474411 state 0x1 ref 1 00000100:00100000:1.0:1789760765.450465:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754748734827 time 754781393965 state 0x0 ref 1 00000100:00100000:1.0:1789760765.450480:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484296448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760765.450526:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23310336, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:1.0:1789760765.450535:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.450550:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:1.0:1789760765.450555:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760765.450562:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23310336->23314431] (req 23310336->23314431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.450588:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23310336->23314431] (req 23310336->23314431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760765.478805:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973009 is committed 00002000:00010000:1.0:1789760765.478886:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23310336 -> 23314432 00010000:00010000:1.0:1789760765.478900:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23310336->23314431] (req 23310336->23314431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.478913:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23310336->23314431] (req 23310336->23314431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.478995:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23310336->23314431] (req 23310336->23314431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.479009:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.479014:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23310336->23314431] (req 23310336->23314431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.479024:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23310336->23314431] (req 23310336->23314431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.479043:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23310336->23314431] (req 23310336->23314431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.479051:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620955a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23310336->23314431] (req 23310336->23314431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.479169:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484296448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28687us (29168us total) trans 4294973009 rc 0/0 00000100:00100000:1.0:1789760765.479179:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5720 00000100:00100000:1.0:1789760765.479188:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754748734827 time 754781393965 state 0x2 ref 0 00000100:00100000:1.0:1789760765.483685:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760765.483902:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484296576 00000100:00100000:2.0:1789760765.483913:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484296576 00000100:00100000:2.0:1789760765.483977:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754748734827 time 754781393965 state 0x0 ref 1 00000100:00100000:2.0:1789760765.483994:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754781654381 time 754781393965 state 0x1 ref 1 00000100:00100000:2.0:1789760765.484027:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754781654381 time 754814958451 state 0x0 ref 1 00000100:00100000:2.0:1789760765.484041:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484296576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760765.484358:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23314432, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:2.0:1789760765.484368:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.484384:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:2.0:1789760765.484389:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760765.484396:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23314432->23318527] (req 23314432->23318527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.484409:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23314432->23318527] (req 23314432->23318527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760765.537882:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973010 is committed 00002000:00010000:1.0:1789760765.537973:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23314432 -> 23318528 00010000:00010000:1.0:1789760765.537987:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23314432->23318527] (req 23314432->23318527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.537999:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23314432->23318527] (req 23314432->23318527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.538014:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23314432->23318527] (req 23314432->23318527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.538022:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.538027:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23314432->23318527] (req 23314432->23318527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.538036:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23314432->23318527] (req 23314432->23318527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.538055:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23314432->23318527] (req 23314432->23318527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.538062:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209561 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23314432->23318527] (req 23314432->23318527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.538206:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484296576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54163us (54526us total) trans 4294973010 rc 0/0 00000100:00100000:1.0:1789760765.538217:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5721 00000100:00100000:1.0:1789760765.538234:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754781654381 time 754814958451 state 0x2 ref 0 00000100:00100000:2.0:1789760765.541691:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760765.541799:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484296704 00000100:00100000:1.0:1789760765.541811:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484296704 00000100:00100000:1.0:1789760765.541864:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754781654381 time 754814958451 state 0x0 ref 1 00000100:00100000:1.0:1789760765.541879:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754815218867 time 754814958451 state 0x1 ref 1 00000100:00100000:1.0:1789760765.541913:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754815218867 time 754872843856 state 0x0 ref 1 00000100:00100000:1.0:1789760765.541933:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484296704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760765.541987:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23318528, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:1.0:1789760765.541996:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.542011:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:1.0:1789760765.542016:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760765.542022:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23318528->23322623] (req 23318528->23322623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.542036:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23318528->23322623] (req 23318528->23322623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760765.585881:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973011 is committed 00002000:00010000:1.0:1789760765.586044:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23318528 -> 23322624 00010000:00010000:1.0:1789760765.586061:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23318528->23322623] (req 23318528->23322623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.586097:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23318528->23322623] (req 23318528->23322623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.586107:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23318528->23322623] (req 23318528->23322623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.586115:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.586120:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23318528->23322623] (req 23318528->23322623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.586130:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23318528->23322623] (req 23318528->23322623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.586161:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23318528->23322623] (req 23318528->23322623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.586169:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209568 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23318528->23322623] (req 23318528->23322623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.586300:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484296704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44371us (44613us total) trans 4294973011 rc 0/0 00000100:00100000:1.0:1789760765.586311:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5722 00000100:00100000:1.0:1789760765.586323:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754815218867 time 754872843856 state 0x2 ref 0 00000100:00100000:2.0:1789760765.590965:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760765.591132:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484296832 00000100:00100000:0.0:1789760765.591159:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484296832 00000100:00100000:0.0:1789760765.591235:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754815218867 time 754872843856 state 0x0 ref 1 00000100:00100000:0.0:1789760765.591251:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754873104272 time 754872843856 state 0x1 ref 1 00000100:00100000:0.0:1789760765.591280:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754873104272 time 754922210134 state 0x0 ref 1 00000100:00100000:0.0:1789760765.591295:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484296832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760765.591348:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23322624, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:0.0:1789760765.591357:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.591376:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:0.0:1789760765.591381:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760765.591387:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23322624->23326719] (req 23322624->23326719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.591401:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23322624->23326719] (req 23322624->23326719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760765.630188:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973012 is committed 00002000:00010000:1.0:1789760765.630367:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23322624 -> 23326720 00010000:00010000:1.0:1789760765.630381:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23322624->23326719] (req 23322624->23326719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.630393:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23322624->23326719] (req 23322624->23326719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.630402:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23322624->23326719] (req 23322624->23326719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.630415:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.630420:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23322624->23326719] (req 23322624->23326719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.630429:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23322624->23326719] (req 23322624->23326719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.630510:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23322624->23326719] (req 23322624->23326719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.630518:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620956f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23322624->23326719] (req 23322624->23326719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.630631:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484296832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39335us (39672us total) trans 4294973012 rc 0/0 00000100:00100000:1.0:1789760765.630641:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5723 00000100:00100000:1.0:1789760765.630650:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754873104272 time 754922210134 state 0x2 ref 0 00000100:00100000:1.0:1789760765.637617:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760765.637711:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484296960 00000100:00100000:3.0:1789760765.637721:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484296960 00000100:00100000:3.0:1789760765.637955:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754873104272 time 754922210134 state 0x0 ref 1 00000100:00100000:3.0:1789760765.637970:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754922470550 time 754922210134 state 0x1 ref 1 00000100:00100000:3.0:1789760765.638000:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754922470550 time 754968931540 state 0x0 ref 1 00000100:00100000:3.0:1789760765.638013:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484296960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760765.638058:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23326720, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:3.0:1789760765.638946:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.638964:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=754s v=(0 0) 00010000:00010000:3.0:1789760765.638968:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760765.638974:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23326720->23330815] (req 23326720->23330815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.638988:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23326720->23330815] (req 23326720->23330815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760765.680779:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23326720 -> 23330816 00010000:00010000:3.0:1789760765.680799:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23326720->23330815] (req 23326720->23330815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.680811:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23326720->23330815] (req 23326720->23330815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.680825:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23326720->23330815] (req 23326720->23330815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.680832:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760765.680837:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23326720->23330815] (req 23326720->23330815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.680846:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23326720->23330815] (req 23326720->23330815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.680867:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23326720->23330815] (req 23326720->23330815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.680876:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209576 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23326720->23330815] (req 23326720->23330815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760765.680976:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484296960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42959us (43364us total) trans 4294973013 rc 0/0 00000100:00100000:3.0:1789760765.680986:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5724 00000100:00100000:3.0:1789760765.680995:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754922470550 time 754968931540 state 0x2 ref 0 00000001:00080000:1.0:1789760765.681809:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973013 is committed 00000100:00100000:2.0:1789760765.691118:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760765.691281:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484297088 00000100:00100000:1.0:1789760765.691292:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484297088 00000100:00100000:1.0:1789760765.691347:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754922470550 time 754968931540 state 0x0 ref 1 00000100:00100000:1.0:1789760765.691364:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754969191956 time 754968931540 state 0x1 ref 1 00000100:00100000:1.0:1789760765.691402:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754969191956 time 755022332658 state 0x0 ref 1 00000100:00100000:1.0:1789760765.691416:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706d850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484297088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760765.691465:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23330816, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:1.0:1789760765.691474:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.691490:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:1.0:1789760765.691493:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760765.691500:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23330816->23334911] (req 23330816->23334911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.691524:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23330816->23334911] (req 23330816->23334911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760765.727933:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973014 is committed 00002000:00010000:3.0:1789760765.728143:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23330816 -> 23334912 00010000:00010000:3.0:1789760765.728160:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23330816->23334911] (req 23330816->23334911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.728179:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23330816->23334911] (req 23330816->23334911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.728189:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23330816->23334911] (req 23330816->23334911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.728196:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760765.728201:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23330816->23334911] (req 23330816->23334911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.728210:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23330816->23334911] (req 23330816->23334911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.728229:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23330816->23334911] (req 23330816->23334911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.728237:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620957d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23330816->23334911] (req 23330816->23334911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760765.728368:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706d850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484297088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36951us (37300us total) trans 4294973014 rc 0/0 00000100:00100000:3.0:1789760765.728378:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5725 00000100:00100000:3.0:1789760765.728387:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754969191956 time 755022332658 state 0x2 ref 0 00000100:00100000:2.0:1789760765.732629:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760765.737658:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484297216 00000100:00100000:1.0:1789760765.737675:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484297216 00000100:00100000:1.0:1789760765.737739:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 754969191956 time 755022332658 state 0x0 ref 1 00000100:00100000:1.0:1789760765.737756:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755022593074 time 755022332658 state 0x1 ref 1 00000100:00100000:1.0:1789760765.737791:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755022593074 time 755068721816 state 0x0 ref 1 00000100:00100000:1.0:1789760765.737804:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484297216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760765.737857:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23334912, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:1.0:1789760765.737866:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.737884:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:1.0:1789760765.737888:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760765.737894:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23334912->23339007] (req 23334912->23339007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.737908:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23334912->23339007] (req 23334912->23339007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760765.764891:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23334912 -> 23339008 00010000:00010000:3.0:1789760765.764911:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23334912->23339007] (req 23334912->23339007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.764925:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23334912->23339007] (req 23334912->23339007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.764935:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23334912->23339007] (req 23334912->23339007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.764943:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760765.764947:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23334912->23339007] (req 23334912->23339007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760765.764949:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973015 is committed 00010000:00010000:3.0:1789760765.764957:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23334912->23339007] (req 23334912->23339007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.764976:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23334912->23339007] (req 23334912->23339007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.764985:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209584 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23334912->23339007] (req 23334912->23339007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760765.765110:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484297216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27303us (32484us total) trans 4294973015 rc 0/0 00000100:00100000:3.0:1789760765.765120:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5726 00000100:00100000:3.0:1789760765.765130:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755022593074 time 755068721816 state 0x2 ref 0 00000100:00100000:1.0:1789760765.770921:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760765.771286:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484297344 00000100:00100000:0.0:1789760765.771297:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484297344 00000100:00100000:2.0:1789760765.771779:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755022593074 time 755068721816 state 0x0 ref 1 00000100:00100000:2.0:1789760765.771798:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755068982232 time 755068721816 state 0x1 ref 1 00000100:00100000:2.0:1789760765.771829:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755068982232 time 755102760731 state 0x0 ref 1 00000100:00100000:2.0:1789760765.771842:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484297344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760765.771893:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23339008, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:2.0:1789760765.771901:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.771916:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:2.0:1789760765.771920:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760765.771926:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23339008->23343103] (req 23339008->23343103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.771938:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23339008->23343103] (req 23339008->23343103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760765.803255:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23339008 -> 23343104 00010000:00010000:1.0:1789760765.803274:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23339008->23343103] (req 23339008->23343103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.803286:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23339008->23343103] (req 23339008->23343103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.803295:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23339008->23343103] (req 23339008->23343103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.803303:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760765.803308:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23339008->23343103] (req 23339008->23343103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.803317:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23339008->23343103] (req 23339008->23343103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.803337:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23339008->23343103] (req 23339008->23343103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.803344:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620958b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23339008->23343103] (req 23339008->23343103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760765.803433:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484297344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31590us (32516us total) trans 4294973016 rc 0/0 00000100:00100000:1.0:1789760765.803445:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5727 00000100:00100000:1.0:1789760765.803455:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755068982232 time 755102760731 state 0x2 ref 0 00000001:00080000:3.0:1789760765.804722:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973016 is committed 00000100:00100000:0.0:1789760765.810662:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760765.810772:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484297472 00000100:00100000:3.0:1789760765.810782:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484297472 00000100:00100000:3.0:1789760765.810834:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755068982232 time 755102760731 state 0x0 ref 1 00000100:00100000:3.0:1789760765.810848:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755103021147 time 755102760731 state 0x1 ref 1 00000100:00100000:3.0:1789760765.810871:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755103021147 time 755141802819 state 0x0 ref 1 00000100:00100000:3.0:1789760765.810882:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484297472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760765.810940:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23343104, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:3.0:1789760765.810949:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.810966:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:3.0:1789760765.810971:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760765.810976:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23343104->23347199] (req 23343104->23347199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.810989:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23343104->23347199] (req 23343104->23347199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.836649:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:2.0:1789760765.863848:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23343104 -> 23347200 00010000:00010000:2.0:1789760765.863868:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23343104->23347199] (req 23343104->23347199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.863881:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23343104->23347199] (req 23343104->23347199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.863892:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23343104->23347199] (req 23343104->23347199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.863909:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00000001:00080000:0.0:1789760765.863914:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973017 is committed 00010000:00010000:2.0:1789760765.863965:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23343104->23347199] (req 23343104->23347199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.863978:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23343104->23347199] (req 23343104->23347199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.864000:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23343104->23347199] (req 23343104->23347199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760765.864008:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209592 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23343104->23347199] (req 23343104->23347199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760765.864237:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484297472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53351us (53579us total) trans 4294973017 rc 0/0 00000100:00100000:2.0:1789760765.864247:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5728 00000100:00100000:2.0:1789760765.864257:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755103021147 time 755141802819 state 0x2 ref 0 00000100:00100000:0.0:1789760765.867764:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760765.867862:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484297600 00000100:00100000:3.0:1789760765.867872:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484297600 00000100:00100000:3.0:1789760765.867987:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755103021147 time 755141802819 state 0x0 ref 1 00000100:00100000:3.0:1789760765.868003:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755142063235 time 755141802819 state 0x1 ref 1 00000100:00100000:3.0:1789760765.868033:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755142063235 time 755198963299 state 0x0 ref 1 00000100:00100000:3.0:1789760765.868046:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484297600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760765.868117:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23347200, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:3.0:1789760765.868127:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.868154:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:3.0:1789760765.868158:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760765.868165:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23347200->23351295] (req 23347200->23351295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.868179:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23347200->23351295] (req 23347200->23351295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760765.892771:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973018 is committed 00002000:00010000:0.0:1789760765.893021:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23347200 -> 23351296 00010000:00010000:0.0:1789760765.893037:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23347200->23351295] (req 23347200->23351295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.893051:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23347200->23351295] (req 23347200->23351295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.893059:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23347200->23351295] (req 23347200->23351295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.893086:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760765.893091:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23347200->23351295] (req 23347200->23351295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.893102:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23347200->23351295] (req 23347200->23351295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.893120:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23347200->23351295] (req 23347200->23351295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.893129:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209599 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23347200->23351295] (req 23347200->23351295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760765.893233:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484297600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25185us (25474us total) trans 4294973018 rc 0/0 00000100:00100000:0.0:1789760765.893243:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5729 00000100:00100000:0.0:1789760765.893253:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755142063235 time 755198963299 state 0x2 ref 0 00000100:00100000:1.0:1789760765.901684:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760765.901769:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484297728 00000100:00100000:3.0:1789760765.901779:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484297728 00000100:00100000:3.0:1789760765.901828:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755142063235 time 755198963299 state 0x0 ref 1 00000100:00100000:3.0:1789760765.901842:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755199223715 time 755198963299 state 0x1 ref 1 00000100:00100000:3.0:1789760765.901873:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755199223715 time 755232804750 state 0x0 ref 1 00000100:00100000:3.0:1789760765.901885:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484297728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760765.901985:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23351296, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:3.0:1789760765.901995:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.902011:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:3.0:1789760765.902015:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760765.902021:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23351296->23355391] (req 23351296->23355391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760765.902035:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23351296->23355391] (req 23351296->23355391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760765.930127:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973019 is committed 00002000:00010000:0.0:1789760765.930237:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23351296 -> 23355392 00010000:00010000:0.0:1789760765.930252:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23351296->23355391] (req 23351296->23355391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.930266:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23351296->23355391] (req 23351296->23355391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.930275:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23351296->23355391] (req 23351296->23355391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.930283:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760765.930288:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23351296->23355391] (req 23351296->23355391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.930297:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23351296->23355391] (req 23351296->23355391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.930325:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23351296->23355391] (req 23351296->23355391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760765.930339:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962095a0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23351296->23355391] (req 23351296->23355391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760765.930425:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484297728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28536us (28745us total) trans 4294973019 rc 0/0 00000100:00100000:0.0:1789760765.930436:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5730 00000100:00100000:0.0:1789760765.930446:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755199223715 time 755232804750 state 0x2 ref 0 00000100:00100000:2.0:1789760765.933510:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760765.933623:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484297856 00000100:00100000:1.0:1789760765.933632:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484297856 00000100:00100000:1.0:1789760765.934015:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755199223715 time 755232804750 state 0x0 ref 1 00000100:00100000:1.0:1789760765.934031:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755233065166 time 755232804750 state 0x1 ref 1 00000100:00100000:1.0:1789760765.934059:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755233065166 time 755264990553 state 0x0 ref 1 00000100:00100000:1.0:1789760765.934220:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484297856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760765.934273:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23355392, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:1.0:1789760765.934283:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.934373:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:1.0:1789760765.934379:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760765.934386:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23355392->23359487] (req 23355392->23359487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760765.934401:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23355392->23359487] (req 23355392->23359487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760766.013363:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23355392 -> 23359488 00010000:00010000:2.0:1789760766.013385:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23355392->23359487] (req 23355392->23359487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.013397:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23355392->23359487] (req 23355392->23359487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.013410:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23355392->23359487] (req 23355392->23359487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.013417:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760766.013421:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23355392->23359487] (req 23355392->23359487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.013430:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23355392->23359487] (req 23355392->23359487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.013452:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23355392->23359487] (req 23355392->23359487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.013459:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095a7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23355392->23359487] (req 23355392->23359487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760766.013554:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484297856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79334us (80047us total) trans 4294973020 rc 0/0 00000100:00100000:2.0:1789760766.013564:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5731 00000100:00100000:2.0:1789760766.013572:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755233065166 time 755264990553 state 0x2 ref 0 00000001:00080000:1.0:1789760766.019444:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973020 is committed 00000100:00100000:1.0:1789760766.020029:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760766.020555:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484297984 00000100:00100000:2.0:1789760766.020565:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484297984 00000100:00080000:3.0:1789760766.020628:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760766.020634:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755233065166 time 755264990553 state 0x0 ref 1 00000100:00100000:2.0:1789760766.020649:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755265250969 time 755264990553 state 0x1 ref 1 00000100:00100000:2.0:1789760766.020670:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755265250969 time 755351601029 state 0x0 ref 1 00000100:00100000:2.0:1789760766.020681:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484297984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760766.020727:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23359488, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:2.0:1789760766.020736:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.020751:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:2.0:1789760766.020755:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760766.020760:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23359488->23363583] (req 23359488->23363583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.020773:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23359488->23363583] (req 23359488->23363583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760766.080533:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973021 is committed 00002000:00010000:2.0:1789760766.080652:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23359488 -> 23363584 00010000:00010000:2.0:1789760766.080667:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23359488->23363583] (req 23359488->23363583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.080680:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23359488->23363583] (req 23359488->23363583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.080689:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23359488->23363583] (req 23359488->23363583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.080696:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760766.080701:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23359488->23363583] (req 23359488->23363583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.080709:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23359488->23363583] (req 23359488->23363583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.080727:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23359488->23363583] (req 23359488->23363583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.080735:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095ae lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23359488->23363583] (req 23359488->23363583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760766.080863:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484297984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60179us (60834us total) trans 4294973021 rc 0/0 00000100:00100000:2.0:1789760766.080874:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5732 00000100:00100000:2.0:1789760766.080883:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755265250969 time 755351601029 state 0x2 ref 0 00000100:00100000:1.0:1789760766.085256:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.085389:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484298112 00000100:00100000:3.0:1789760766.085400:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484298112 00000100:00100000:3.0:1789760766.085460:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755265250969 time 755351601029 state 0x0 ref 1 00000100:00100000:3.0:1789760766.085478:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755351861445 time 755351601029 state 0x1 ref 1 00000100:00100000:3.0:1789760766.085508:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755351861445 time 755416438034 state 0x0 ref 1 00000100:00100000:3.0:1789760766.085522:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484298112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.085575:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23363584, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:3.0:1789760766.085585:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.085608:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:3.0:1789760766.085613:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.085619:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23363584->23367679] (req 23363584->23367679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.085635:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23363584->23367679] (req 23363584->23367679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760766.134409:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973022 is committed 00002000:00010000:3.0:1789760766.134514:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23363584 -> 23367680 00010000:00010000:3.0:1789760766.134529:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23363584->23367679] (req 23363584->23367679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.134541:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23363584->23367679] (req 23363584->23367679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.134550:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23363584->23367679] (req 23363584->23367679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.134557:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.134562:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23363584->23367679] (req 23363584->23367679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.134570:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23363584->23367679] (req 23363584->23367679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.134588:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23363584->23367679] (req 23363584->23367679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.134596:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095b5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23363584->23367679] (req 23363584->23367679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.136341:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484298112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49177us (49449us total) trans 4294973022 rc 0/0 00000100:00100000:3.0:1789760766.136365:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5733 00000100:00100000:3.0:1789760766.136376:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755351861445 time 755416438034 state 0x2 ref 0 00000100:00100000:0.0:1789760766.138974:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760766.139037:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484298240 00000100:00100000:1.0:1789760766.139054:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484298240 00000100:00100000:1.0:1789760766.139123:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755351861445 time 755416438034 state 0x0 ref 1 00000100:00100000:1.0:1789760766.139149:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755416698450 time 755416438034 state 0x1 ref 1 00000100:00100000:1.0:1789760766.139177:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755416698450 time 755470108201 state 0x0 ref 1 00000100:00100000:1.0:1789760766.139189:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484298240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760766.139244:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23367680, mtime: 0x6aad94fd, atime: 0x0, ctime: 0x6aad94fd, blocks: 0xb807 00010000:00010000:1.0:1789760766.139252:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.139273:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:1.0:1789760766.139277:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760766.139284:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23367680->23371775] (req 23367680->23371775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.139297:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23367680->23371775] (req 23367680->23371775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760766.192566:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973023 is committed 00002000:00010000:3.0:1789760766.192781:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23367680 -> 23371776 00002000:00010000:3.0:1789760766.192786:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760765 -> 1789760766 00002000:00010000:3.0:1789760766.192789:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760765 -> 1789760766 00010000:00010000:3.0:1789760766.192802:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23367680->23371775] (req 23367680->23371775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.192814:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23367680->23371775] (req 23367680->23371775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.192822:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23367680->23371775] (req 23367680->23371775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.192829:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.192834:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23367680->23371775] (req 23367680->23371775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.192843:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23367680->23371775] (req 23367680->23371775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.192862:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23367680->23371775] (req 23367680->23371775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.192870:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf38263962095bc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23367680->23371775] (req 23367680->23371775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.193057:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484298240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53865us (54088us total) trans 4294973023 rc 0/0 00000100:00100000:3.0:1789760766.193114:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5734 00000100:00100000:3.0:1789760766.193124:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755416698450 time 755470108201 state 0x2 ref 0 00000100:00100000:1.0:1789760766.198406:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760766.198503:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484298368 00000100:00100000:2.0:1789760766.198513:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484298368 00000100:00100000:2.0:1789760766.198557:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755416698450 time 755470108201 state 0x0 ref 1 00000100:00100000:2.0:1789760766.198567:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755470368617 time 755470108201 state 0x1 ref 1 00000100:00100000:2.0:1789760766.198590:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755470368617 time 755529522436 state 0x0 ref 1 00000100:00100000:2.0:1789760766.198598:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484298368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760766.198636:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23371776, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:2.0:1789760766.198713:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.198724:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:2.0:1789760766.198728:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760766.198734:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23371776->23375871] (req 23371776->23375871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.198747:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23371776->23375871] (req 23371776->23375871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760766.237298:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973024 is committed 00002000:00010000:3.0:1789760766.237402:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23371776 -> 23375872 00010000:00010000:3.0:1789760766.237422:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23371776->23375871] (req 23371776->23375871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.237436:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23371776->23375871] (req 23371776->23375871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.237446:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23371776->23375871] (req 23371776->23375871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.237523:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.237529:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23371776->23375871] (req 23371776->23375871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.237543:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23371776->23375871] (req 23371776->23375871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.237562:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23371776->23375871] (req 23371776->23375871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.237569:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962095c3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23371776->23375871] (req 23371776->23375871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.237662:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484298368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39059us (39260us total) trans 4294973024 rc 0/0 00000100:00100000:3.0:1789760766.237672:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5735 00000100:00100000:3.0:1789760766.237682:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755470368617 time 755529522436 state 0x2 ref 0 00000100:00100000:1.0:1789760766.242249:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.243154:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484298496 00000100:00100000:3.0:1789760766.243165:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484298496 00000100:00100000:2.0:1789760766.243300:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755470368617 time 755529522436 state 0x0 ref 1 00000100:00100000:2.0:1789760766.243315:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755529782852 time 755529522436 state 0x1 ref 1 00000100:00100000:2.0:1789760766.243376:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755529782852 time 755574306610 state 0x0 ref 1 00000100:00100000:2.0:1789760766.243390:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484298496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760766.243438:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23375872, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:2.0:1789760766.243447:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.243534:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:2.0:1789760766.243539:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760766.243545:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23375872->23379967] (req 23375872->23379967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.243565:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23375872->23379967] (req 23375872->23379967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760766.290907:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23375872 -> 23379968 00010000:00010000:2.0:1789760766.290938:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23375872->23379967] (req 23375872->23379967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.290952:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23375872->23379967] (req 23375872->23379967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.290961:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23375872->23379967] (req 23375872->23379967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.290968:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760766.290973:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23375872->23379967] (req 23375872->23379967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.290983:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23375872->23379967] (req 23375872->23379967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.291004:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23375872->23379967] (req 23375872->23379967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.291012:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf38263962095ca lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23375872->23379967] (req 23375872->23379967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760766.291263:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484298496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47871us (49018us total) trans 4294973025 rc 0/0 00000100:00100000:2.0:1789760766.291273:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5736 00000100:00100000:2.0:1789760766.291282:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755529782852 time 755574306610 state 0x2 ref 0 00000001:00080000:3.0:1789760766.291822:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973025 is committed 00000100:00100000:0.0:1789760766.295294:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.295453:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484298624 00000100:00100000:3.0:1789760766.295462:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484298624 00000100:00100000:3.0:1789760766.295515:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755529782852 time 755574306610 state 0x0 ref 1 00000100:00100000:3.0:1789760766.295531:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755574567026 time 755574306610 state 0x1 ref 1 00000100:00100000:3.0:1789760766.295565:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755574567026 time 755626495817 state 0x0 ref 1 00000100:00100000:3.0:1789760766.295578:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484298624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.295629:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23379968, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.295638:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.295654:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:3.0:1789760766.295658:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.295665:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23379968->23384063] (req 23379968->23384063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.295679:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23379968->23384063] (req 23379968->23384063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760766.332297:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973026 is committed 00002000:00010000:1.0:1789760766.332495:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23379968 -> 23384064 00010000:00010000:1.0:1789760766.332510:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23379968->23384063] (req 23379968->23384063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.332521:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23379968->23384063] (req 23379968->23384063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.332531:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23379968->23384063] (req 23379968->23384063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.332538:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760766.332543:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23379968->23384063] (req 23379968->23384063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.332551:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23379968->23384063] (req 23379968->23384063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.332571:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23379968->23384063] (req 23379968->23384063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.332579:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962095d1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23379968->23384063] (req 23379968->23384063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760766.332680:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484298624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37098us (37390us total) trans 4294973026 rc 0/0 00000100:00100000:1.0:1789760766.332690:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5737 00000100:00100000:1.0:1789760766.332700:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755574567026 time 755626495817 state 0x2 ref 0 00000100:00100000:2.0:1789760766.341452:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.341525:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484298752 00000100:00100000:3.0:1789760766.341535:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484298752 00000100:00100000:3.0:1789760766.341591:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755574567026 time 755626495817 state 0x0 ref 1 00000100:00100000:3.0:1789760766.341607:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755626756233 time 755626495817 state 0x1 ref 1 00000100:00100000:3.0:1789760766.341634:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755626756233 time 755672565099 state 0x0 ref 1 00000100:00100000:3.0:1789760766.341646:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484298752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.341691:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23384064, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.341700:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.341720:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:3.0:1789760766.341724:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.341731:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23384064->23388159] (req 23384064->23388159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.341745:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23384064->23388159] (req 23384064->23388159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760766.379955:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973027 is committed 00002000:00010000:0.0:1789760766.380298:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23384064 -> 23388160 00010000:00010000:0.0:1789760766.380313:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23384064->23388159] (req 23384064->23388159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.380327:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23384064->23388159] (req 23384064->23388159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.380336:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23384064->23388159] (req 23384064->23388159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.380344:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760766.380348:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23384064->23388159] (req 23384064->23388159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.380357:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23384064->23388159] (req 23384064->23388159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.380378:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23384064->23388159] (req 23384064->23388159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.380385:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962095d8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23384064->23388159] (req 23384064->23388159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760766.380481:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484298752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38832us (39033us total) trans 4294973027 rc 0/0 00000100:00100000:0.0:1789760766.380492:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5738 00000100:00100000:0.0:1789760766.380502:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755626756233 time 755672565099 state 0x2 ref 0 00000100:00100000:0.0:1789760766.385385:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760766.385462:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484298880 00000100:00100000:2.0:1789760766.385472:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484298880 00000100:00100000:3.0:1789760766.385857:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755626756233 time 755672565099 state 0x0 ref 1 00000100:00100000:3.0:1789760766.385874:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755672825515 time 755672565099 state 0x1 ref 1 00000100:00100000:3.0:1789760766.385905:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755672825515 time 755716835467 state 0x0 ref 1 00000100:00100000:3.0:1789760766.385926:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484298880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.385971:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23388160, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.385979:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.385994:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:3.0:1789760766.385998:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.386004:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23388160->23392255] (req 23388160->23392255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.386024:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23388160->23392255] (req 23388160->23392255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760766.434862:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23388160 -> 23392256 00010000:00010000:3.0:1789760766.434883:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23388160->23392255] (req 23388160->23392255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.434896:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23388160->23392255] (req 23388160->23392255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.434905:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23388160->23392255] (req 23388160->23392255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.434981:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.434986:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23388160->23392255] (req 23388160->23392255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.434996:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23388160->23392255] (req 23388160->23392255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.435016:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23388160->23392255] (req 23388160->23392255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.435024:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095df lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23388160->23392255] (req 23388160->23392255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.435291:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484298880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49363us (49908us total) trans 4294973028 rc 0/0 00000100:00100000:3.0:1789760766.435302:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5739 00000100:00100000:3.0:1789760766.435311:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755672825515 time 755716835467 state 0x2 ref 0 00000001:00080000:0.0:1789760766.436318:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973028 is committed 00000100:00100000:1.0:1789760766.438490:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760766.439734:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484299008 00000100:00100000:2.0:1789760766.439748:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484299008 00000100:00100000:2.0:1789760766.439809:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755672825515 time 755716835467 state 0x0 ref 1 00000100:00100000:2.0:1789760766.439825:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755717095883 time 755716835467 state 0x1 ref 1 00000100:00100000:2.0:1789760766.439949:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755717095883 time 755770878059 state 0x0 ref 1 00000100:00100000:2.0:1789760766.439964:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484299008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760766.440014:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23392256, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:2.0:1789760766.440031:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.440047:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:2.0:1789760766.440051:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760766.440057:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23392256->23396351] (req 23392256->23396351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.440103:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23392256->23396351] (req 23392256->23396351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760766.472746:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973029 is committed 00002000:00010000:0.0:1789760766.472902:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23392256 -> 23396352 00010000:00010000:0.0:1789760766.472976:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23392256->23396351] (req 23392256->23396351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.472990:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23392256->23396351] (req 23392256->23396351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.472999:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23392256->23396351] (req 23392256->23396351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.473005:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760766.473009:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23392256->23396351] (req 23392256->23396351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.473017:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23392256->23396351] (req 23392256->23396351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.473176:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23392256->23396351] (req 23392256->23396351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.473185:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf38263962095e6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23392256->23396351] (req 23392256->23396351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760766.473314:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484299008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33348us (34828us total) trans 4294973029 rc 0/0 00000100:00100000:0.0:1789760766.473325:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5740 00000100:00100000:0.0:1789760766.473336:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755717095883 time 755770878059 state 0x2 ref 0 00000100:00100000:2.0:1789760766.479265:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760766.479334:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484299136 00000100:00100000:1.0:1789760766.479345:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484299136 00000100:00100000:1.0:1789760766.479400:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755717095883 time 755770878059 state 0x0 ref 1 00000100:00100000:1.0:1789760766.479415:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755771138475 time 755770878059 state 0x1 ref 1 00000100:00100000:1.0:1789760766.479439:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755771138475 time 755810370385 state 0x0 ref 1 00000100:00100000:1.0:1789760766.479451:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484299136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760766.479495:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23396352, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:1.0:1789760766.479504:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.479525:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:1.0:1789760766.479529:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760766.479535:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23396352->23400447] (req 23396352->23400447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.479548:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23396352->23400447] (req 23396352->23400447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760766.509090:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23396352 -> 23400448 00010000:00010000:3.0:1789760766.509109:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23396352->23400447] (req 23396352->23400447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.509122:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23396352->23400447] (req 23396352->23400447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.509141:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23396352->23400447] (req 23396352->23400447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.509149:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.509154:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23396352->23400447] (req 23396352->23400447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.509162:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23396352->23400447] (req 23396352->23400447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.509186:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23396352->23400447] (req 23396352->23400447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.509199:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962095ed lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23396352->23400447] (req 23396352->23400447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.509294:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484299136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29839us (30033us total) trans 4294973030 rc 0/0 00000100:00100000:3.0:1789760766.509304:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5741 00000100:00100000:3.0:1789760766.509313:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755771138475 time 755810370385 state 0x2 ref 0 00000001:00080000:2.0:1789760766.510432:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973030 is committed 00000100:00100000:0.0:1789760766.517213:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.517304:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484299264 00000100:00100000:3.0:1789760766.517314:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484299264 00000100:00100000:3.0:1789760766.517369:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755771138475 time 755810370385 state 0x0 ref 1 00000100:00100000:3.0:1789760766.517384:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755810630801 time 755810370385 state 0x1 ref 1 00000100:00100000:3.0:1789760766.517450:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755810630801 time 755848380214 state 0x0 ref 1 00000100:00100000:3.0:1789760766.517463:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9651c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484299264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.517511:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23400448, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.517520:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.517535:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:3.0:1789760766.517540:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.517545:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23400448->23404543] (req 23400448->23404543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.517564:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23400448->23404543] (req 23400448->23404543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760766.557500:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973031 is committed 00002000:00010000:2.0:1789760766.557721:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23400448 -> 23404544 00010000:00010000:2.0:1789760766.557735:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23400448->23404543] (req 23400448->23404543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.557746:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23400448->23404543] (req 23400448->23404543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.557754:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23400448->23404543] (req 23400448->23404543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.557761:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760766.557765:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23400448->23404543] (req 23400448->23404543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.557774:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23400448->23404543] (req 23400448->23404543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.557794:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23400448->23404543] (req 23400448->23404543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.557807:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf38263962095f4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23400448->23404543] (req 23400448->23404543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760766.557901:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9651c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484299264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40436us (40691us total) trans 4294973031 rc 0/0 00000100:00100000:2.0:1789760766.557919:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5742 00000100:00100000:2.0:1789760766.557929:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755810630801 time 755848380214 state 0x2 ref 0 00000100:00100000:1.0:1789760766.563760:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.563977:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484299392 00000100:00100000:3.0:1789760766.563987:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484299392 00000100:00100000:3.0:1789760766.564037:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755810630801 time 755848380214 state 0x0 ref 1 00000100:00100000:3.0:1789760766.564053:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755848640630 time 755848380214 state 0x1 ref 1 00000100:00100000:3.0:1789760766.564120:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755848640630 time 755895050361 state 0x0 ref 1 00000100:00100000:3.0:1789760766.564142:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484299392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.564190:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23404544, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.564197:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.564212:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:3.0:1789760766.564216:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.564223:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23404544->23408639] (req 23404544->23408639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.564236:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23404544->23408639] (req 23404544->23408639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760766.595230:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973032 is committed 00002000:00010000:3.0:1789760766.595429:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23404544 -> 23408640 00010000:00010000:3.0:1789760766.595444:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23404544->23408639] (req 23404544->23408639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.595457:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23404544->23408639] (req 23404544->23408639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.595466:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23404544->23408639] (req 23404544->23408639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.595472:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.595477:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23404544->23408639] (req 23404544->23408639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.595486:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23404544->23408639] (req 23404544->23408639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.595504:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23404544->23408639] (req 23404544->23408639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.595512:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf38263962095fb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23404544->23408639] (req 23404544->23408639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.595635:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484299392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31500us (31880us total) trans 4294973032 rc 0/0 00000100:00100000:3.0:1789760766.595646:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5743 00000100:00100000:3.0:1789760766.595656:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755848640630 time 755895050361 state 0x2 ref 0 00000100:00100000:1.0:1789760766.608436:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760766.608530:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484299520 00000100:00100000:0.0:1789760766.608540:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484299520 00000100:00100000:0.0:1789760766.608594:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755848640630 time 755895050361 state 0x0 ref 1 00000100:00100000:0.0:1789760766.608609:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755895310777 time 755895050361 state 0x1 ref 1 00000100:00100000:0.0:1789760766.608687:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755895310777 time 755939617094 state 0x0 ref 1 00000100:00100000:0.0:1789760766.608701:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484299520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760766.608759:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23408640, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:0.0:1789760766.608768:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.608781:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:0.0:1789760766.608786:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760766.608792:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23408640->23412735] (req 23408640->23412735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.608805:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23408640->23412735] (req 23408640->23412735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760766.642506:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973033 is committed 00002000:00010000:3.0:1789760766.642827:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23408640 -> 23412736 00010000:00010000:3.0:1789760766.642841:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23408640->23412735] (req 23408640->23412735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.642851:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23408640->23412735] (req 23408640->23412735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.642857:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23408640->23412735] (req 23408640->23412735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.642861:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.642865:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23408640->23412735] (req 23408640->23412735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.642872:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23408640->23412735] (req 23408640->23412735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.642886:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23408640->23412735] (req 23408640->23412735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.642890:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209602 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23408640->23412735] (req 23408640->23412735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.642976:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484299520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34274us (34546us total) trans 4294973033 rc 0/0 00000100:00100000:3.0:1789760766.642987:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5744 00000100:00100000:3.0:1789760766.642996:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755895310777 time 755939617094 state 0x2 ref 0 00000100:00100000:0.0:1789760766.646309:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760766.646644:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484299648 00000100:00100000:1.0:1789760766.646653:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484299648 00000100:00100000:1.0:1789760766.646709:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755895310777 time 755939617094 state 0x0 ref 1 00000100:00100000:1.0:1789760766.646724:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755939877510 time 755939617094 state 0x1 ref 1 00000100:00100000:1.0:1789760766.646783:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755939877510 time 755977713925 state 0x0 ref 1 00000100:00100000:1.0:1789760766.646796:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484299648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760766.646843:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23412736, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:1.0:1789760766.646851:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.646866:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=755s v=(0 0) 00010000:00010000:1.0:1789760766.646870:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760766.646877:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23412736->23416831] (req 23412736->23416831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.646889:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23412736->23416831] (req 23412736->23416831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760766.672289:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23412736 -> 23416832 00010000:00010000:3.0:1789760766.672307:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23412736->23416831] (req 23412736->23416831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.672316:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23412736->23416831] (req 23412736->23416831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.672322:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23412736->23416831] (req 23412736->23416831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.672326:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.672330:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23412736->23416831] (req 23412736->23416831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.672336:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23412736->23416831] (req 23412736->23416831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.672352:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23412736->23416831] (req 23412736->23416831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.672357:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209609 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23412736->23416831] (req 23412736->23416831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.672432:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484299648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25633us (26126us total) trans 4294973034 rc 0/0 00000100:00100000:3.0:1789760766.672446:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5745 00000100:00100000:3.0:1789760766.672456:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755939877510 time 755977713925 state 0x2 ref 0 00000001:00080000:2.0:1789760766.672524:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973034 is committed 00000100:00100000:1.0:1789760766.676656:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760766.676968:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484299776 00000100:00100000:2.0:1789760766.676978:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484299776 00000100:00100000:2.0:1789760766.677031:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755939877510 time 755977713925 state 0x0 ref 1 00000100:00100000:2.0:1789760766.677046:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755977974341 time 755977713925 state 0x1 ref 1 00000100:00100000:2.0:1789760766.679233:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755977974341 time 756010152481 state 0x0 ref 1 00000100:00100000:2.0:1789760766.679261:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484299776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760766.679316:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23416832, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:2.0:1789760766.679325:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.679340:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:2.0:1789760766.679346:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760766.679351:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23416832->23420927] (req 23416832->23420927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.679365:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23416832->23420927] (req 23416832->23420927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760766.719140:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973035 is committed 00002000:00010000:2.0:1789760766.719344:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23416832 -> 23420928 00010000:00010000:2.0:1789760766.719358:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23416832->23420927] (req 23416832->23420927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.719372:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23416832->23420927] (req 23416832->23420927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.719382:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23416832->23420927] (req 23416832->23420927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.719390:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760766.719395:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23416832->23420927] (req 23416832->23420927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.719405:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23416832->23420927] (req 23416832->23420927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.719425:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23416832->23420927] (req 23416832->23420927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.719434:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209610 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23416832->23420927] (req 23416832->23420927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760766.719539:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484299776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40285us (42887us total) trans 4294973035 rc 0/0 00000100:00100000:2.0:1789760766.719550:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5746 00000100:00100000:2.0:1789760766.719559:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755977974341 time 756010152481 state 0x2 ref 0 00000100:00100000:1.0:1789760766.723515:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.723647:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484299904 00000100:00100000:3.0:1789760766.723657:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484299904 00000100:00100000:3.0:1789760766.723714:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 755977974341 time 756010152481 state 0x0 ref 1 00000100:00100000:3.0:1789760766.723731:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756010412897 time 756010152481 state 0x1 ref 1 00000100:00100000:3.0:1789760766.723762:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756010412897 time 756054693430 state 0x0 ref 1 00000100:00100000:3.0:1789760766.723775:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484299904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.723832:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23420928, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.723841:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.723857:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760766.723862:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.723869:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23420928->23425023] (req 23420928->23425023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.723884:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23420928->23425023] (req 23420928->23425023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760766.751841:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973036 is committed 00002000:00010000:2.0:1789760766.756846:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23420928 -> 23425024 00010000:00010000:2.0:1789760766.756867:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23420928->23425023] (req 23420928->23425023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.756882:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23420928->23425023] (req 23420928->23425023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.756891:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23420928->23425023] (req 23420928->23425023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.756898:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760766.756902:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23420928->23425023] (req 23420928->23425023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.756922:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23420928->23425023] (req 23420928->23425023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.756944:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23420928->23425023] (req 23420928->23425023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.756957:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209617 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23420928->23425023] (req 23420928->23425023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760766.757059:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484299904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33283us (33549us total) trans 4294973036 rc 0/0 00000100:00100000:2.0:1789760766.757104:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5747 00000100:00100000:2.0:1789760766.757114:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756010412897 time 756054693430 state 0x2 ref 0 00000100:00100000:1.0:1789760766.760795:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.761442:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484300032 00000100:00100000:3.0:1789760766.761452:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484300032 00000100:00100000:3.0:1789760766.761506:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756010412897 time 756054693430 state 0x0 ref 1 00000100:00100000:3.0:1789760766.761533:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756054953846 time 756054693430 state 0x1 ref 1 00000100:00100000:3.0:1789760766.761564:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756054953846 time 756092494806 state 0x0 ref 1 00000100:00100000:3.0:1789760766.761577:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484300032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.761954:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23425024, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.761965:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.761981:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760766.761986:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.761991:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23425024->23429119] (req 23425024->23429119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.762004:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23425024->23429119] (req 23425024->23429119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760766.801684:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973037 is committed 00002000:00010000:0.0:1789760766.801794:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23425024 -> 23429120 00010000:00010000:0.0:1789760766.801809:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23425024->23429119] (req 23425024->23429119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.801821:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23425024->23429119] (req 23425024->23429119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.801831:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23425024->23429119] (req 23425024->23429119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.801838:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760766.801843:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23425024->23429119] (req 23425024->23429119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.801852:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23425024->23429119] (req 23425024->23429119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.801872:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23425024->23429119] (req 23425024->23429119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.801879:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620961e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23425024->23429119] (req 23425024->23429119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760766.802040:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484300032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40460us (41248us total) trans 4294973037 rc 0/0 00000100:00100000:0.0:1789760766.802051:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5748 00000100:00100000:0.0:1789760766.802060:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756054953846 time 756092494806 state 0x2 ref 0 00000100:00100000:1.0:1789760766.807000:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.807195:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484300160 00000100:00100000:3.0:1789760766.807205:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484300160 00000100:00100000:3.0:1789760766.807268:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756054953846 time 756092494806 state 0x0 ref 1 00000100:00100000:3.0:1789760766.807282:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756092755222 time 756092494806 state 0x1 ref 1 00000100:00100000:3.0:1789760766.807310:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756092755222 time 756138241150 state 0x0 ref 1 00000100:00100000:3.0:1789760766.807325:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484300160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.807370:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23429120, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.807379:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.807395:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760766.807400:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.807405:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23429120->23433215] (req 23429120->23433215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.807419:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23429120->23433215] (req 23429120->23433215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760766.845354:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973038 is committed 00002000:00010000:3.0:1789760766.845521:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23429120 -> 23433216 00010000:00010000:3.0:1789760766.845537:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23429120->23433215] (req 23429120->23433215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.845549:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23429120->23433215] (req 23429120->23433215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.845558:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23429120->23433215] (req 23429120->23433215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.845565:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.845570:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23429120->23433215] (req 23429120->23433215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.845579:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23429120->23433215] (req 23429120->23433215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.845608:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23429120->23433215] (req 23429120->23433215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.845617:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209625 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23429120->23433215] (req 23429120->23433215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.845715:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484300160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38386us (38720us total) trans 4294973038 rc 0/0 00000100:00100000:3.0:1789760766.845725:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5749 00000100:00100000:3.0:1789760766.845735:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756092755222 time 756138241150 state 0x2 ref 0 00000100:00100000:2.0:1789760766.855405:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.855525:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484300288 00000100:00100000:3.0:1789760766.855535:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484300288 00000100:00100000:3.0:1789760766.855656:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756092755222 time 756138241150 state 0x0 ref 1 00000100:00100000:3.0:1789760766.855672:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756138501566 time 756138241150 state 0x1 ref 1 00000100:00100000:3.0:1789760766.855742:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756138501566 time 756186672560 state 0x0 ref 1 00000100:00100000:3.0:1789760766.855755:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484300288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.855802:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23433216, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.855810:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.855826:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760766.855831:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.855837:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23433216->23437311] (req 23433216->23437311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.855851:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23433216->23437311] (req 23433216->23437311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760766.860325:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760766.896801:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973039 is committed 00002000:00010000:3.0:1789760766.896905:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23433216 -> 23437312 00010000:00010000:3.0:1789760766.896931:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23433216->23437311] (req 23433216->23437311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.896945:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23433216->23437311] (req 23433216->23437311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.896954:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23433216->23437311] (req 23433216->23437311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.896961:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760766.896966:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23433216->23437311] (req 23433216->23437311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.896976:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23433216->23437311] (req 23433216->23437311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.896994:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23433216->23437311] (req 23433216->23437311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.897001:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620962c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23433216->23437311] (req 23433216->23437311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760766.897205:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484300288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41448us (41805us total) trans 4294973039 rc 0/0 00000100:00100000:3.0:1789760766.897223:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5750 00000100:00100000:3.0:1789760766.897234:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756138501566 time 756186672560 state 0x2 ref 0 00000100:00100000:2.0:1789760766.900426:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760766.900510:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484300416 00000100:00100000:3.0:1789760766.900519:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484300416 00000100:00100000:3.0:1789760766.900572:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756138501566 time 756186672560 state 0x0 ref 1 00000100:00100000:3.0:1789760766.900658:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756186932976 time 756186672560 state 0x1 ref 1 00000100:00100000:3.0:1789760766.900690:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756186932976 time 756231620998 state 0x0 ref 1 00000100:00100000:3.0:1789760766.900704:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484300416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.900755:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23437312, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.900764:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.900780:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760766.900785:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.900790:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23437312->23441407] (req 23437312->23441407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.900809:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23437312->23441407] (req 23437312->23441407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760766.936556:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973040 is committed 00002000:00010000:1.0:1789760766.936811:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23437312 -> 23441408 00010000:00010000:1.0:1789760766.936827:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23437312->23441407] (req 23437312->23441407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.936840:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23437312->23441407] (req 23437312->23441407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.936848:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23437312->23441407] (req 23437312->23441407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.936856:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760766.936860:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23437312->23441407] (req 23437312->23441407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.936870:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23437312->23441407] (req 23437312->23441407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.936891:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23437312->23441407] (req 23437312->23441407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760766.936900:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209633 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23437312->23441407] (req 23437312->23441407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760766.937000:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484300416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36295us (36580us total) trans 4294973040 rc 0/0 00000100:00100000:1.0:1789760766.937011:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5751 00000100:00100000:1.0:1789760766.937021:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756186932976 time 756231620998 state 0x2 ref 0 00000100:00100000:2.0:1789760766.945153:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760766.945368:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484300544 00000100:00100000:0.0:1789760766.945379:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484300544 00000100:00100000:0.0:1789760766.945441:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756186932976 time 756231620998 state 0x0 ref 1 00000100:00100000:0.0:1789760766.945456:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756231881414 time 756231620998 state 0x1 ref 1 00000100:00100000:0.0:1789760766.945537:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756231881414 time 756276467172 state 0x0 ref 1 00000100:00100000:0.0:1789760766.945550:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484300544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760766.945607:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23441408, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:0.0:1789760766.945617:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.945634:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:0.0:1789760766.945638:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760766.945645:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23441408->23445503] (req 23441408->23445503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.945660:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23441408->23445503] (req 23441408->23445503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760766.984056:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973041 is committed 00002000:00010000:0.0:1789760766.984225:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23441408 -> 23445504 00010000:00010000:0.0:1789760766.984240:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23441408->23445503] (req 23441408->23445503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.984252:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23441408->23445503] (req 23441408->23445503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.984262:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23441408->23445503] (req 23441408->23445503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.984269:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760766.984274:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23441408->23445503] (req 23441408->23445503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.984284:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23441408->23445503] (req 23441408->23445503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.984303:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23441408->23445503] (req 23441408->23445503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760766.984314:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620963a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23441408->23445503] (req 23441408->23445503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760766.984411:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484300544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38860us (39263us total) trans 4294973041 rc 0/0 00000100:00100000:0.0:1789760766.984422:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5752 00000100:00100000:0.0:1789760766.984431:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756231881414 time 756276467172 state 0x2 ref 0 00000100:00100000:3.0:1789760766.991027:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760766.992334:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484300672 00000100:00100000:2.0:1789760766.992345:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484300672 00000100:00100000:3.0:1789760766.992554:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756231881414 time 756276467172 state 0x0 ref 1 00000100:00100000:3.0:1789760766.992571:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756276727588 time 756276467172 state 0x1 ref 1 00000100:00100000:3.0:1789760766.992603:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756276727588 time 756323534822 state 0x0 ref 1 00000100:00100000:3.0:1789760766.992616:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484300672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760766.992661:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23445504, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760766.992670:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.992686:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760766.992691:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760766.992697:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23445504->23449599] (req 23445504->23449599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760766.992711:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23445504->23449599] (req 23445504->23449599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760767.033061:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973042 is committed 00002000:00010000:1.0:1789760767.033157:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23445504 -> 23449600 00010000:00010000:1.0:1789760767.033187:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23445504->23449599] (req 23445504->23449599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.033199:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23445504->23449599] (req 23445504->23449599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.033208:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23445504->23449599] (req 23445504->23449599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.033223:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760767.033229:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23445504->23449599] (req 23445504->23449599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.033239:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23445504->23449599] (req 23445504->23449599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.033260:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23445504->23449599] (req 23445504->23449599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.033269:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209641 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23445504->23449599] (req 23445504->23449599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760767.033359:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484300672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40741us (42336us total) trans 4294973042 rc 0/0 00000100:00100000:1.0:1789760767.033370:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5753 00000100:00100000:1.0:1789760767.033379:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756276727588 time 756323534822 state 0x2 ref 0 00000100:00100000:1.0:1789760767.038946:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760767.039104:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484300800 00000100:00100000:3.0:1789760767.039115:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484300800 00000100:00100000:3.0:1789760767.039190:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756276727588 time 756323534822 state 0x0 ref 1 00000100:00100000:3.0:1789760767.039206:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756323795238 time 756323534822 state 0x1 ref 1 00000100:00100000:3.0:1789760767.039238:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756323795238 time 756370168468 state 0x0 ref 1 00000100:00100000:3.0:1789760767.039252:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484300800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760767.039299:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23449600, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:3.0:1789760767.039309:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.039323:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760767.039328:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760767.039334:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23449600->23453695] (req 23449600->23453695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.039349:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23449600->23453695] (req 23449600->23453695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760767.059801:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973043 is committed 00002000:00010000:3.0:1789760767.060913:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23449600 -> 23453696 00010000:00010000:3.0:1789760767.060930:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23449600->23453695] (req 23449600->23453695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.060943:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23449600->23453695] (req 23449600->23453695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.060958:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23449600->23453695] (req 23449600->23453695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.060966:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760767.060971:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23449600->23453695] (req 23449600->23453695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.060980:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23449600->23453695] (req 23449600->23453695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.060999:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23449600->23453695] (req 23449600->23453695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.061007:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209648 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23449600->23453695] (req 23449600->23453695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760767.061363:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484300800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 22108us (22420us total) trans 4294973043 rc 0/0 00000100:00100000:3.0:1789760767.061373:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5754 00000100:00100000:3.0:1789760767.061382:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756323795238 time 756370168468 state 0x2 ref 0 00000100:00100000:2.0:1789760767.065658:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760767.065728:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484300928 00000100:00100000:1.0:1789760767.065737:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484300928 00000100:00100000:1.0:1789760767.065786:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756323795238 time 756370168468 state 0x0 ref 1 00000100:00100000:1.0:1789760767.065799:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756370428884 time 756370168468 state 0x1 ref 1 00000100:00100000:1.0:1789760767.065821:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756370428884 time 756396752380 state 0x0 ref 1 00000100:00100000:1.0:1789760767.065830:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484300928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760767.065873:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23453696, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:1.0:1789760767.065881:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.065895:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:1.0:1789760767.065899:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760767.065904:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23453696->23457791] (req 23453696->23457791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.065916:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23453696->23457791] (req 23453696->23457791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760767.109350:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973044 is committed 00002000:00010000:1.0:1789760767.109607:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23453696 -> 23457792 00010000:00010000:1.0:1789760767.109621:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23453696->23457791] (req 23453696->23457791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.109634:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23453696->23457791] (req 23453696->23457791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.109643:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23453696->23457791] (req 23453696->23457791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.109651:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760767.109655:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23453696->23457791] (req 23453696->23457791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.109664:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23453696->23457791] (req 23453696->23457791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.109689:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23453696->23457791] (req 23453696->23457791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.109697:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620964f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23453696->23457791] (req 23453696->23457791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760767.109785:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484300928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43951us (44131us total) trans 4294973044 rc 0/0 00000100:00100000:1.0:1789760767.109796:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5755 00000100:00100000:1.0:1789760767.109804:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756370428884 time 756396752380 state 0x2 ref 0 00000100:00100000:2.0:1789760767.116994:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760767.117139:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484301056 00000100:00100000:0.0:1789760767.117149:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484301056 00000100:00100000:0.0:1789760767.117217:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756370428884 time 756396752380 state 0x0 ref 1 00000100:00100000:0.0:1789760767.117234:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756397012796 time 756396752380 state 0x1 ref 1 00000100:00100000:0.0:1789760767.117265:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756397012796 time 756448196294 state 0x0 ref 1 00000100:00100000:0.0:1789760767.117280:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484301056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760767.117328:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23457792, mtime: 0x6aad94fe, atime: 0x0, ctime: 0x6aad94fe, blocks: 0xb807 00010000:00010000:0.0:1789760767.117336:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.117350:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:0.0:1789760767.117355:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760767.117361:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23457792->23461887] (req 23457792->23461887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.117374:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23457792->23461887] (req 23457792->23461887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760767.138669:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973045 is committed 00002000:00010000:0.0:1789760767.138730:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23457792 -> 23461888 00002000:00010000:0.0:1789760767.138735:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760766 -> 1789760767 00002000:00010000:0.0:1789760767.138737:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760766 -> 1789760767 00010000:00010000:0.0:1789760767.138751:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23457792->23461887] (req 23457792->23461887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.138764:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23457792->23461887] (req 23457792->23461887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.138773:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23457792->23461887] (req 23457792->23461887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.138781:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760767.138786:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23457792->23461887] (req 23457792->23461887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.138795:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23457792->23461887] (req 23457792->23461887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.138815:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23457792->23461887] (req 23457792->23461887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.138821:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209656 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23457792->23461887] (req 23457792->23461887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760767.138904:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484301056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 21623us (21915us total) trans 4294973045 rc 0/0 00000100:00100000:0.0:1789760767.138914:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5756 00000100:00100000:0.0:1789760767.138923:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756397012796 time 756448196294 state 0x2 ref 0 00000100:00100000:0.0:1789760767.141797:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760767.142021:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484301184 00000100:00100000:3.0:1789760767.142034:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484301184 00000100:00100000:3.0:1789760767.142349:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756397012796 time 756448196294 state 0x0 ref 1 00000100:00100000:3.0:1789760767.142364:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756448456710 time 756448196294 state 0x1 ref 1 00000100:00100000:1.0:1789760767.144586:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756448456710 time 756475453591 state 0x0 ref 1 00000100:00100000:1.0:1789760767.144615:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9652300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484301184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760767.144748:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23461888, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:1.0:1789760767.144757:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.144778:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:1.0:1789760767.144782:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760767.144788:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23461888->23465983] (req 23461888->23465983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.144802:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23461888->23465983] (req 23461888->23465983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760767.176233:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973046 is committed 00002000:00010000:1.0:1789760767.176396:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23461888 -> 23465984 00010000:00010000:1.0:1789760767.176412:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23461888->23465983] (req 23461888->23465983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.176425:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23461888->23465983] (req 23461888->23465983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.176433:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23461888->23465983] (req 23461888->23465983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.176439:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760767.176444:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23461888->23465983] (req 23461888->23465983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.176452:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23461888->23465983] (req 23461888->23465983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.176470:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23461888->23465983] (req 23461888->23465983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.176476:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620965d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23461888->23465983] (req 23461888->23465983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760767.176568:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9652300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484301184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31960us (34775us total) trans 4294973046 rc 0/0 00000100:00100000:1.0:1789760767.176582:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5757 00000100:00100000:1.0:1789760767.176592:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756448456710 time 756475453591 state 0x2 ref 0 00000100:00100000:2.0:1789760767.181979:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760767.182158:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484301312 00000100:00100000:3.0:1789760767.182178:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484301312 00000100:00100000:3.0:1789760767.182233:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756448456710 time 756475453591 state 0x0 ref 1 00000100:00100000:3.0:1789760767.182248:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756475714007 time 756475453591 state 0x1 ref 1 00000100:00100000:3.0:1789760767.182343:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756475714007 time 756513272833 state 0x0 ref 1 00000100:00100000:3.0:1789760767.182357:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484301312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760767.182404:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23465984, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:3.0:1789760767.182414:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.182429:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760767.182434:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760767.182441:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23465984->23470079] (req 23465984->23470079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.182455:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23465984->23470079] (req 23465984->23470079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760767.213333:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23465984 -> 23470080 00010000:00010000:3.0:1789760767.213354:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23465984->23470079] (req 23465984->23470079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.213366:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23465984->23470079] (req 23465984->23470079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.213375:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23465984->23470079] (req 23465984->23470079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.213383:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760767.213388:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23465984->23470079] (req 23465984->23470079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.213396:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23465984->23470079] (req 23465984->23470079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.213413:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23465984->23470079] (req 23465984->23470079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.213419:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209664 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23465984->23470079] (req 23465984->23470079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760767.213512:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484301312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31152us (31538us total) trans 4294973047 rc 0/0 00000100:00100000:3.0:1789760767.213523:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5758 00000100:00100000:3.0:1789760767.213532:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756475714007 time 756513272833 state 0x2 ref 0 00000001:00080000:1.0:1789760767.214029:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973047 is committed 00000100:00100000:1.0:1789760767.221447:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760767.221633:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484301440 00000100:00100000:3.0:1789760767.221644:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484301440 00000100:00100000:3.0:1789760767.221701:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756475714007 time 756513272833 state 0x0 ref 1 00000100:00100000:3.0:1789760767.221716:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756513533249 time 756513272833 state 0x1 ref 1 00000100:00100000:3.0:1789760767.221797:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756513533249 time 756552727363 state 0x0 ref 1 00000100:00100000:3.0:1789760767.221810:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484301440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760767.221858:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23470080, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:3.0:1789760767.221866:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.221881:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760767.221886:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760767.221892:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23470080->23474175] (req 23470080->23474175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.221920:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23470080->23474175] (req 23470080->23474175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760767.257964:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23470080 -> 23474176 00010000:00010000:3.0:1789760767.257984:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23470080->23474175] (req 23470080->23474175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.257997:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23470080->23474175] (req 23470080->23474175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.258007:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23470080->23474175] (req 23470080->23474175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.258015:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760767.258020:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23470080->23474175] (req 23470080->23474175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.258030:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23470080->23474175] (req 23470080->23474175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.258051:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23470080->23474175] (req 23470080->23474175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.258059:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620966b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23470080->23474175] (req 23470080->23474175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760767.258186:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484301440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36374us (36742us total) trans 4294973048 rc 0/0 00000100:00100000:3.0:1789760767.258196:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5759 00000100:00100000:3.0:1789760767.258205:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756513533249 time 756552727363 state 0x2 ref 0 00000001:00080000:1.0:1789760767.258880:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973048 is committed 00000100:00100000:2.0:1789760767.263347:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760767.263452:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484301568 00000100:00100000:0.0:1789760767.263462:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484301568 00000100:00100000:0.0:1789760767.263521:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756513533249 time 756552727363 state 0x0 ref 1 00000100:00100000:0.0:1789760767.263536:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756552987779 time 756552727363 state 0x1 ref 1 00000100:00100000:0.0:1789760767.263567:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756552987779 time 756594497523 state 0x0 ref 1 00000100:00100000:0.0:1789760767.263581:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484301568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760767.263691:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23474176, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:0.0:1789760767.263701:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.263717:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:0.0:1789760767.263721:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760767.263727:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23474176->23478271] (req 23474176->23478271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.263741:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23474176->23478271] (req 23474176->23478271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760767.305384:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973049 is committed 00002000:00010000:1.0:1789760767.305469:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23474176 -> 23478272 00010000:00010000:1.0:1789760767.305485:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23474176->23478271] (req 23474176->23478271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.305499:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23474176->23478271] (req 23474176->23478271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.305507:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23474176->23478271] (req 23474176->23478271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.305514:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760767.305519:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23474176->23478271] (req 23474176->23478271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.305528:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23474176->23478271] (req 23474176->23478271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.305548:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23474176->23478271] (req 23474176->23478271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.305561:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209672 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23474176->23478271] (req 23474176->23478271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760767.305693:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484301568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42111us (42350us total) trans 4294973049 rc 0/0 00000100:00100000:1.0:1789760767.305704:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5760 00000100:00100000:1.0:1789760767.305714:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756552987779 time 756594497523 state 0x2 ref 0 00000100:00100000:0.0:1789760767.311406:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760767.311889:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484301696 00000100:00100000:2.0:1789760767.311899:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484301696 00000100:00100000:2.0:1789760767.312186:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756552987779 time 756594497523 state 0x0 ref 1 00000100:00100000:2.0:1789760767.312203:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756594757939 time 756594497523 state 0x1 ref 1 00000100:00100000:2.0:1789760767.312241:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756594757939 time 756643171900 state 0x0 ref 1 00000100:00100000:2.0:1789760767.312254:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9653b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484301696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760767.312308:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23478272, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:2.0:1789760767.312318:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.312334:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:2.0:1789760767.312339:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760767.312345:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23478272->23482367] (req 23478272->23482367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.312359:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23478272->23482367] (req 23478272->23482367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760767.368110:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973050 is committed 00002000:00010000:2.0:1789760767.370421:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23478272 -> 23482368 00010000:00010000:2.0:1789760767.370442:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23478272->23482367] (req 23478272->23482367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.370457:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23478272->23482367] (req 23478272->23482367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.370467:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23478272->23482367] (req 23478272->23482367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.370476:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760767.370481:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23478272->23482367] (req 23478272->23482367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.370492:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23478272->23482367] (req 23478272->23482367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.370514:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23478272->23482367] (req 23478272->23482367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.370522:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209679 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23478272->23482367] (req 23478272->23482367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760767.370957:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9653b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484301696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58699us (59555us total) trans 4294973050 rc 0/0 00000100:00100000:2.0:1789760767.370968:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5761 00000100:00100000:2.0:1789760767.370978:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756594757939 time 756643171900 state 0x2 ref 0 00000100:00100000:0.0:1789760767.372541:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508932352:0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.372602:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760767.372829:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9651f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3479:1876699508931968:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.372865:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760767.372920:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508932480:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.372942:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760767.372960:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508932992:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.372979:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760767.372994:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508933504:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.373010:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760767.373042:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508932480 00000100:00100000:1.0:1789760767.373049:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508932480 00000100:00100000:1.0:1789760767.373111:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 459 00000100:00100000:1.0:1789760767.373130:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508932480:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.373189:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508932480:12345-0@lo:400:kworker.0 Request processed in 70us (245us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760767.373195:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 459 00000100:00100000:2.0:1789760767.373317:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508931968 00000100:00100000:2.0:1789760767.373322:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508931968 00000100:00100000:2.0:1789760767.373349:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 488 00000100:00100000:2.0:1789760767.373355:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c803b450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3479:x1876699508931968:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760767.373389:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c803b450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3479:x1876699508931968:12345-0@lo:400:kworker.0 Request processed in 33us (523us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760767.373394:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 488 00000100:00100000:1.0:1789760767.373445:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508932992 00000100:00100000:1.0:1789760767.373449:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508932992 00000100:00100000:1.0:1789760767.373486:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508933504 00000100:00100000:1.0:1789760767.373489:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508933504 00000100:00100000:1.0:1789760767.373505:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 337 00000100:00100000:1.0:1789760767.373512:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508932992:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.373542:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508932992:12345-0@lo:400:kworker.0 Request processed in 31us (561us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760767.373547:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 337 00000100:00100000:1.0:1789760767.373557:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 338 00000100:00100000:1.0:1789760767.373561:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508933504:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.373584:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508933504:12345-0@lo:400:kworker.0 Request processed in 22us (573us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760767.373589:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 338 00000100:00100000:0.0:1789760767.373776:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508932864:0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.373810:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760767.373828:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508933376:0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.373849:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760767.373877:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508932096:0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.373900:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760767.373917:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508932608:0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.373936:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760767.373950:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508933120:0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.373966:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760767.373980:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df829180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508933632:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760767.373998:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760767.374013:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [3->2] 00000100:00100000:0.0:1789760767.374018:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508932224:0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.374037:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760767.374052:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508932736:0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.374093:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760767.374111:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508933248:0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.374142:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760767.374172:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508932352 00000100:00100000:0.0:1789760767.374176:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508932352 00000100:00100000:0.0:1789760767.374205:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508932096 00000100:00100000:0.0:1789760767.374208:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508932096 00000100:00080000:2.0:1789760767.374316:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:2.0:1789760767.374321:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1789760767.374361:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9651f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3479:1876699508931968:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.374380:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508932480:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.374391:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508932992:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.374400:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508933504:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.374432:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508933632 00000100:00100000:1.0:1789760767.374436:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508933632 00000100:00100000:0.0:1789760767.374503:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508932608 00000100:00100000:0.0:1789760767.374507:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508932608 00000100:00100000:3.0:1789760767.374711:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508932864 00000100:00100000:3.0:1789760767.374716:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508932864 00000100:00100000:1.0:1789760767.374765:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 400 00000100:00100000:1.0:1789760767.374772:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508933632:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:1.0:1789760767.374804:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760767.374849:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508933632:12345-0@lo:13:osp-pre-1-1.0 Request processed in 77us (849us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760767.374855:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 400 00000100:00100000:1.0:1789760767.374875:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508932224 00000100:00100000:1.0:1789760767.374879:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508932224 00000100:00100000:1.0:1789760767.374900:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 460 00000100:00100000:1.0:1789760767.374906:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508932224:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.374938:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508932224:12345-0@lo:400:kworker.0 Request processed in 32us (899us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760767.374943:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 460 00000100:00100000:1.0:1789760767.374961:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508933376 00000100:00100000:1.0:1789760767.374964:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508933376 00000100:00100000:1.0:1789760767.374979:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 339 00000100:00100000:1.0:1789760767.374984:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508933376:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.375017:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508933376:12345-0@lo:400:kworker.0 Request processed in 33us (1167us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760767.375023:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 339 00000100:00100000:1.0:1789760767.375052:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508933376:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.375098:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508932224:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.375112:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df829180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508933632:0@lo:13:osp-pre-1-1.0 00000100:00080000:2.0:1789760767.375799:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:2.0:1789760767.375806:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:2.0:1789760767.375809:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:0.0:1789760767.375890:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508933120 00000100:00100000:0.0:1789760767.375896:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508933120 00000100:00100000:0.0:1789760767.375931:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508932736 00000100:00100000:0.0:1789760767.375934:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508932736 00000100:00100000:0.0:1789760767.375946:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508933248 00000100:00100000:0.0:1789760767.375948:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508933248 00000100:00100000:0.0:1789760767.375964:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 965 00000100:00100000:0.0:1789760767.375971:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508932352:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.376025:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508932352:12345-0@lo:400:kworker.0 Request processed in 54us (3423us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760767.376032:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 965 00000100:00100000:0.0:1789760767.376046:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 966 00000100:00100000:0.0:1789760767.376054:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508932096:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.376107:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508932096:12345-0@lo:400:kworker.0 Request processed in 54us (2205us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760767.376113:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 966 00000100:00100000:0.0:1789760767.376135:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 967 00000100:00100000:0.0:1789760767.376143:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508932608:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.376172:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b82680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508932608:12345-0@lo:400:kworker.0 Request processed in 31us (2235us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760767.376178:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 967 00000100:00100000:0.0:1789760767.376189:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 968 00000100:00100000:0.0:1789760767.376196:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b81500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508933120:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.376333:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b81500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508933120:12345-0@lo:400:kworker.0 Request processed in 137us (2364us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760767.376339:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 968 00000100:00100000:0.0:1789760767.376353:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 969 00000100:00100000:0.0:1789760767.376361:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defde300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508932736:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.376388:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defde300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508932736:12345-0@lo:400:kworker.0 Request processed in 29us (2295us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760767.376394:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 969 00000100:00100000:0.0:1789760767.376405:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 970 00000100:00100000:0.0:1789760767.376413:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defded80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508933248:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760767.376451:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defded80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508933248:12345-0@lo:400:kworker.0 Request processed in 40us (2308us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760767.376456:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 970 00000100:00100000:1.0:1789760767.376563:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508932352:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.376596:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508932736:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.376608:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508933248:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.376617:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508932096:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.376800:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508932608:0@lo:400:kworker.0 00000100:00100000:1.0:1789760767.376812:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508933120:0@lo:400:kworker.0 00000100:00080000:2.0:1789760767.376899:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760767.376913:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:1.0:1789760767.377770:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1789760767.377788:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5116a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508933760:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760767.377841:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760767.377879:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508933760 00000100:00100000:1.0:1789760767.377885:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508933760 00000100:00100000:1.0:1789760767.377997:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 401 00000100:00100000:1.0:1789760767.378006:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508933760:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:1.0:1789760767.378027:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760767.378084:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508933760:12345-0@lo:13:osp-pre-1-0.0 Request processed in 61us (224us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760767.378091:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 401 00000100:00100000:1.0:1789760767.378142:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5116a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508933760:0@lo:13:osp-pre-1-0.0 00000100:00100000:3.0:1789760767.379422:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 340 00000100:00100000:3.0:1789760767.379434:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508932864:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760767.379501:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0ca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508932864:12345-0@lo:400:kworker.0 Request processed in 70us (5689us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760767.379507:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 340 00000100:00100000:1.0:1789760767.379706:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508932864:0@lo:400:kworker.0 00000100:00100000:2.0:1789760767.379894:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760767.380457:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484301824 00000100:00100000:3.0:1789760767.380465:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484301824 00000100:00100000:3.0:1789760767.380509:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756594757939 time 756643171900 state 0x0 ref 1 00000100:00100000:3.0:1789760767.380527:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756643432316 time 756643171900 state 0x1 ref 1 00000100:00100000:3.0:1789760767.380553:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756643432316 time 756711484556 state 0x0 ref 1 00000100:00100000:3.0:1789760767.380565:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484301824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760767.380609:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23482368, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:3.0:1789760767.380618:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.380634:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760767.380639:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760767.380645:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23482368->23486463] (req 23482368->23486463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.380658:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23482368->23486463] (req 23482368->23486463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760767.413838:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973051 is committed 00002000:00010000:2.0:1789760767.413936:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23482368 -> 23486464 00010000:00010000:2.0:1789760767.413950:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23482368->23486463] (req 23482368->23486463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.413963:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23482368->23486463] (req 23482368->23486463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.413973:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23482368->23486463] (req 23482368->23486463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.413980:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760767.413985:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23482368->23486463] (req 23482368->23486463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.413994:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23482368->23486463] (req 23482368->23486463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.414011:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23482368->23486463] (req 23482368->23486463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.414020:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209680 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23482368->23486463] (req 23482368->23486463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760767.414155:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484301824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33587us (34258us total) trans 4294973051 rc 0/0 00000100:00100000:2.0:1789760767.414169:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5762 00000100:00100000:2.0:1789760767.414178:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756643432316 time 756711484556 state 0x2 ref 0 00000100:00100000:0.0:1789760767.421134:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760767.421381:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484301952 00000100:00100000:3.0:1789760767.421392:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484301952 00000100:00100000:3.0:1789760767.421448:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756643432316 time 756711484556 state 0x0 ref 1 00000100:00100000:3.0:1789760767.421465:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756711744972 time 756711484556 state 0x1 ref 1 00000100:00100000:3.0:1789760767.421527:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756711744972 time 756752457066 state 0x0 ref 1 00000100:00100000:3.0:1789760767.421540:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484301952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760767.421590:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23486464, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:3.0:1789760767.421599:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.421616:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:3.0:1789760767.421621:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760767.421627:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23486464->23490559] (req 23486464->23490559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.421641:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23486464->23490559] (req 23486464->23490559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760767.463652:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23486464 -> 23490560 00010000:00010000:0.0:1789760767.463673:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23486464->23490559] (req 23486464->23490559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.463698:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23486464->23490559] (req 23486464->23490559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.463707:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23486464->23490559] (req 23486464->23490559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.463715:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760767.463720:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23486464->23490559] (req 23486464->23490559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.463729:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23486464->23490559] (req 23486464->23490559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.463747:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23486464->23490559] (req 23486464->23490559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.463756:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209687 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23486464->23490559] (req 23486464->23490559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760767.463849:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484301952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42306us (42730us total) trans 4294973052 rc 0/0 00000100:00100000:0.0:1789760767.463859:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5763 00000100:00100000:0.0:1789760767.463869:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756711744972 time 756752457066 state 0x2 ref 0 00000001:00080000:2.0:1789760767.464099:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973052 is committed 00000100:00100000:1.0:1789760767.467927:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760767.468130:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484302080 00000100:00100000:0.0:1789760767.468141:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484302080 00000100:00100000:0.0:1789760767.468198:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756711744972 time 756752457066 state 0x0 ref 1 00000100:00100000:0.0:1789760767.468223:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756752717482 time 756752457066 state 0x1 ref 1 00000100:00100000:0.0:1789760767.468258:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756752717482 time 756799189097 state 0x0 ref 1 00000100:00100000:0.0:1789760767.468277:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484302080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760767.468325:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23490560, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:0.0:1789760767.468334:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.468350:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:0.0:1789760767.468354:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760767.468360:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23490560->23494655] (req 23490560->23494655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.468375:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23490560->23494655] (req 23490560->23494655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760767.520759:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23490560 -> 23494656 00010000:00010000:1.0:1789760767.520781:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23490560->23494655] (req 23490560->23494655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.520794:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23490560->23494655] (req 23490560->23494655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.520803:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23490560->23494655] (req 23490560->23494655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.520810:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760767.520815:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23490560->23494655] (req 23490560->23494655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.520824:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23490560->23494655] (req 23490560->23494655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.520844:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23490560->23494655] (req 23490560->23494655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.520852:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620968e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23490560->23494655] (req 23490560->23494655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760767.520962:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484302080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52684us (53038us total) trans 4294973053 rc 0/0 00000100:00100000:1.0:1789760767.520974:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5764 00000100:00100000:1.0:1789760767.520983:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756752717482 time 756799189097 state 0x2 ref 0 00000001:00080000:2.0:1789760767.521182:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973053 is committed 00000100:00100000:1.0:1789760767.525718:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760767.525978:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484302208 00000100:00100000:2.0:1789760767.525989:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484302208 00000100:00100000:2.0:1789760767.526217:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756752717482 time 756799189097 state 0x0 ref 1 00000100:00100000:2.0:1789760767.526233:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756799449513 time 756799189097 state 0x1 ref 1 00000100:00100000:2.0:1789760767.526303:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756799449513 time 756857232996 state 0x0 ref 1 00000100:00100000:2.0:1789760767.526319:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484302208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760767.526366:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23494656, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:2.0:1789760767.526374:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.526390:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:2.0:1789760767.526394:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760767.526400:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23494656->23498751] (req 23494656->23498751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.526414:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23494656->23498751] (req 23494656->23498751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760767.561479:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23494656 -> 23498752 00010000:00010000:3.0:1789760767.561500:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23494656->23498751] (req 23494656->23498751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.561512:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23494656->23498751] (req 23494656->23498751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.561522:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23494656->23498751] (req 23494656->23498751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.561528:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760767.561533:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23494656->23498751] (req 23494656->23498751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.561541:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23494656->23498751] (req 23494656->23498751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.561560:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23494656->23498751] (req 23494656->23498751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760767.561568:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209695 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23494656->23498751] (req 23494656->23498751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760767.561658:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484302208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35338us (35945us total) trans 4294973054 rc 0/0 00000100:00100000:3.0:1789760767.561667:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5765 00000100:00100000:3.0:1789760767.561677:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756799449513 time 756857232996 state 0x2 ref 0 00000001:00080000:0.0:1789760767.562852:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973054 is committed 00000100:00100000:0.0:1789760767.566376:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760767.566479:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484302336 00000100:00100000:2.0:1789760767.566489:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484302336 00000100:00100000:2.0:1789760767.566540:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756799449513 time 756857232996 state 0x0 ref 1 00000100:00100000:2.0:1789760767.566554:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756857493412 time 756857232996 state 0x1 ref 1 00000100:00100000:2.0:1789760767.566585:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756857493412 time 756897515821 state 0x0 ref 1 00000100:00100000:2.0:1789760767.566599:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484302336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760767.566648:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23498752, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:2.0:1789760767.566657:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.566672:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:2.0:1789760767.566676:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760767.566682:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23498752->23502847] (req 23498752->23502847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.566696:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23498752->23502847] (req 23498752->23502847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760767.601405:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973055 is committed 00002000:00010000:0.0:1789760767.601491:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23498752 -> 23502848 00010000:00010000:0.0:1789760767.601509:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23498752->23502847] (req 23498752->23502847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.601520:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23498752->23502847] (req 23498752->23502847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.601529:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23498752->23502847] (req 23498752->23502847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.601536:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760767.601541:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23498752->23502847] (req 23498752->23502847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.601550:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23498752->23502847] (req 23498752->23502847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.601569:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23498752->23502847] (req 23498752->23502847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.601576:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620969c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23498752->23502847] (req 23498752->23502847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760767.601698:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484302336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35098us (35326us total) trans 4294973055 rc 0/0 00000100:00100000:0.0:1789760767.601708:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5766 00000100:00100000:0.0:1789760767.601718:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756857493412 time 756897515821 state 0x2 ref 0 00000100:00100000:1.0:1789760767.617202:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760767.621659:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484302464 00000100:00100000:0.0:1789760767.621675:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484302464 00000100:00100000:0.0:1789760767.621748:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756857493412 time 756897515821 state 0x0 ref 1 00000100:00100000:0.0:1789760767.621765:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756897776237 time 756897515821 state 0x1 ref 1 00000100:00100000:0.0:1789760767.621990:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756897776237 time 756952919966 state 0x0 ref 1 00000100:00100000:0.0:1789760767.622017:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484302464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760767.622091:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23502848, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:0.0:1789760767.622102:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.622118:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=756s v=(0 0) 00010000:00010000:0.0:1789760767.622133:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760767.622139:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23502848->23506943] (req 23502848->23506943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.622152:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23502848->23506943] (req 23502848->23506943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760767.690634:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23502848 -> 23506944 00010000:00010000:1.0:1789760767.690652:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23502848->23506943] (req 23502848->23506943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.690665:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23502848->23506943] (req 23502848->23506943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.690678:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23502848->23506943] (req 23502848->23506943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.690685:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760767.690689:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23502848->23506943] (req 23502848->23506943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.690698:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23502848->23506943] (req 23502848->23506943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.690721:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23502848->23506943] (req 23502848->23506943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.690729:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962096a3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23502848->23506943] (req 23502848->23506943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760767.690873:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484302464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 68865us (73675us total) trans 4294973056 rc 0/0 00000100:00100000:1.0:1789760767.690883:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5767 00000100:00100000:1.0:1789760767.690893:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756897776237 time 756952919966 state 0x2 ref 0 00000001:00080000:1.0:1789760767.690942:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973056 is committed 00000100:00100000:1.0:1789760767.702622:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760767.702684:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484302592 00000100:00100000:0.0:1789760767.702695:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484302592 00000100:00100000:0.0:1789760767.702755:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756897776237 time 756952919966 state 0x0 ref 1 00000100:00100000:0.0:1789760767.702769:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756953180382 time 756952919966 state 0x1 ref 1 00000100:00100000:0.0:1789760767.702843:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756953180382 time 757033773000 state 0x0 ref 1 00000100:00100000:0.0:1789760767.702857:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484302592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760767.702916:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23506944, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:0.0:1789760767.702924:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.702938:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:0.0:1789760767.702942:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760767.702948:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23506944->23511039] (req 23506944->23511039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.702960:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23506944->23511039] (req 23506944->23511039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760767.745986:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973057 is committed 00002000:00010000:1.0:1789760767.747646:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23506944 -> 23511040 00010000:00010000:1.0:1789760767.747665:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23506944->23511039] (req 23506944->23511039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.747679:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23506944->23511039] (req 23506944->23511039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.747688:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23506944->23511039] (req 23506944->23511039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.747695:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760767.747700:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23506944->23511039] (req 23506944->23511039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.747709:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23506944->23511039] (req 23506944->23511039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.747731:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23506944->23511039] (req 23506944->23511039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.747738:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962096aa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23506944->23511039] (req 23506944->23511039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760767.747831:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484302592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44972us (45213us total) trans 4294973057 rc 0/0 00000100:00100000:1.0:1789760767.747842:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5768 00000100:00100000:1.0:1789760767.747851:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756953180382 time 757033773000 state 0x2 ref 0 00000100:00100000:1.0:1789760767.751338:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760767.751554:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484302720 00000100:00100000:2.0:1789760767.751565:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484302720 00000100:00100000:2.0:1789760767.751622:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 756953180382 time 757033773000 state 0x0 ref 1 00000100:00100000:2.0:1789760767.751638:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757034033416 time 757033773000 state 0x1 ref 1 00000100:00100000:2.0:1789760767.751666:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757034033416 time 757082597378 state 0x0 ref 1 00000100:00100000:2.0:1789760767.751678:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484302720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760767.751722:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23511040, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:2.0:1789760767.751731:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.751747:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:2.0:1789760767.751751:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760767.751757:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23511040->23515135] (req 23511040->23515135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.751770:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23511040->23515135] (req 23511040->23515135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760767.781592:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23511040 -> 23515136 00010000:00010000:0.0:1789760767.781614:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23511040->23515135] (req 23511040->23515135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.781628:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23511040->23515135] (req 23511040->23515135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.781637:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23511040->23515135] (req 23511040->23515135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.781644:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760767.781648:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23511040->23515135] (req 23511040->23515135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.781657:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23511040->23515135] (req 23511040->23515135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.781676:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23511040->23515135] (req 23511040->23515135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.781684:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf38263962096b1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23511040->23515135] (req 23511040->23515135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760767.781788:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484302720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30106us (30454us total) trans 4294973058 rc 0/0 00000100:00100000:0.0:1789760767.781799:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5769 00000100:00100000:0.0:1789760767.781809:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757034033416 time 757082597378 state 0x2 ref 0 00000001:00080000:1.0:1789760767.781913:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973058 is committed 00000100:00100000:0.0:1789760767.785760:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760767.785965:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484302848 00000100:00100000:2.0:1789760767.785975:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484302848 00000100:00100000:2.0:1789760767.786030:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757034033416 time 757082597378 state 0x0 ref 1 00000100:00100000:2.0:1789760767.786046:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757082857794 time 757082597378 state 0x1 ref 1 00000100:00100000:2.0:1789760767.786114:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757082857794 time 757117044908 state 0x0 ref 1 00000100:00100000:2.0:1789760767.786138:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484302848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760767.786188:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23515136, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:2.0:1789760767.786197:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.786214:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:2.0:1789760767.786218:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760767.786224:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23515136->23519231] (req 23515136->23519231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.786236:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23515136->23519231] (req 23515136->23519231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760767.847094:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23515136 -> 23519232 00010000:00010000:1.0:1789760767.847114:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23515136->23519231] (req 23515136->23519231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.847138:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23515136->23519231] (req 23515136->23519231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.847147:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23515136->23519231] (req 23515136->23519231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.847155:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760767.847159:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23515136->23519231] (req 23515136->23519231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.847169:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23515136->23519231] (req 23515136->23519231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.847188:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23515136->23519231] (req 23515136->23519231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.847196:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962096b8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23515136->23519231] (req 23515136->23519231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760767.847286:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484302848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61146us (61530us total) trans 4294973059 rc 0/0 00000100:00100000:1.0:1789760767.847297:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5770 00000100:00100000:1.0:1789760767.847307:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757082857794 time 757117044908 state 0x2 ref 0 00000001:00080000:1.0:1789760767.847349:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973059 is committed 00000100:00100000:3.0:1789760767.854721:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760767.855444:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484302976 00000100:00100000:1.0:1789760767.855456:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484302976 00000100:00100000:1.0:1789760767.855512:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757082857794 time 757117044908 state 0x0 ref 1 00000100:00100000:1.0:1789760767.855528:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757117305324 time 757117044908 state 0x1 ref 1 00000100:00100000:1.0:1789760767.855569:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757117305324 time 757186500358 state 0x0 ref 1 00000100:00100000:1.0:1789760767.855582:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484302976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760767.855630:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23519232, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:1.0:1789760767.855639:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.855654:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:1.0:1789760767.855659:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760767.855666:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23519232->23523327] (req 23519232->23523327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.855679:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23519232->23523327] (req 23519232->23523327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.884340:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760767.928633:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973060 is committed 00002000:00010000:1.0:1789760767.928828:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23519232 -> 23523328 00010000:00010000:1.0:1789760767.928844:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23519232->23523327] (req 23519232->23523327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.928863:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23519232->23523327] (req 23519232->23523327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.928872:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23519232->23523327] (req 23519232->23523327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.928879:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760767.928884:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23519232->23523327] (req 23519232->23523327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.928893:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23519232->23523327] (req 23519232->23523327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.928921:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23519232->23523327] (req 23519232->23523327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.928928:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf38263962096bf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23519232->23523327] (req 23519232->23523327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760767.929011:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484302976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73426us (74294us total) trans 4294973060 rc 0/0 00000100:00100000:1.0:1789760767.929020:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5771 00000100:00100000:1.0:1789760767.929029:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757117305324 time 757186500358 state 0x2 ref 0 00000100:00100000:2.0:1789760767.934168:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760767.934290:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484303104 00000100:00100000:1.0:1789760767.934306:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484303104 00000100:00100000:1.0:1789760767.934361:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757117305324 time 757186500358 state 0x0 ref 1 00000100:00100000:1.0:1789760767.934376:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757186760774 time 757186500358 state 0x1 ref 1 00000100:00100000:1.0:1789760767.934420:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757186760774 time 757265346385 state 0x0 ref 1 00000100:00100000:1.0:1789760767.934433:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484303104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760767.934482:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23523328, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:1.0:1789760767.934490:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.934506:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:1.0:1789760767.934511:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760767.934518:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23523328->23527423] (req 23523328->23527423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760767.934533:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23523328->23527423] (req 23523328->23527423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760767.976566:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973061 is committed 00002000:00010000:0.0:1789760767.976652:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23523328 -> 23527424 00010000:00010000:0.0:1789760767.976668:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23523328->23527423] (req 23523328->23527423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.976681:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23523328->23527423] (req 23523328->23527423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.976691:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23523328->23527423] (req 23523328->23527423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.976698:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760767.976703:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23523328->23527423] (req 23523328->23527423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.976713:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23523328->23527423] (req 23523328->23527423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.976734:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23523328->23527423] (req 23523328->23527423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760767.976743:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf38263962096c6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23523328->23527423] (req 23523328->23527423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760767.976852:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484303104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42415us (42687us total) trans 4294973061 rc 0/0 00000100:00100000:0.0:1789760767.976863:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5772 00000100:00100000:0.0:1789760767.976874:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757186760774 time 757265346385 state 0x2 ref 0 00000100:00100000:0.0:1789760767.982127:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760767.982214:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484303232 00000100:00100000:2.0:1789760767.982225:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484303232 00000100:00100000:2.0:1789760767.982276:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757186760774 time 757265346385 state 0x0 ref 1 00000100:00100000:2.0:1789760767.982290:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757265606801 time 757265346385 state 0x1 ref 1 00000100:00100000:2.0:1789760767.982312:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757265606801 time 757313243235 state 0x0 ref 1 00000100:00100000:2.0:1789760767.982324:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484303232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760767.982385:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23527424, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:2.0:1789760767.982394:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.982410:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:2.0:1789760767.982413:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760767.982421:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23527424->23531519] (req 23527424->23531519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760767.982435:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23527424->23531519] (req 23527424->23531519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760768.049448:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973062 is committed 00002000:00010000:0.0:1789760768.049553:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23527424 -> 23531520 00010000:00010000:0.0:1789760768.049568:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23527424->23531519] (req 23527424->23531519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.049581:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23527424->23531519] (req 23527424->23531519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.049590:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23527424->23531519] (req 23527424->23531519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.049596:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760768.049601:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23527424->23531519] (req 23527424->23531519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.049610:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23527424->23531519] (req 23527424->23531519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.049644:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23527424->23531519] (req 23527424->23531519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.049652:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf38263962096cd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23527424->23531519] (req 23527424->23531519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760768.049743:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484303232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67417us (67676us total) trans 4294973062 rc 0/0 00000100:00100000:0.0:1789760768.049752:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5773 00000100:00100000:0.0:1789760768.049761:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757265606801 time 757313243235 state 0x2 ref 0 00000100:00100000:1.0:1789760768.056687:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.056777:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484303360 00000100:00100000:3.0:1789760768.056794:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484303360 00000100:00100000:3.0:1789760768.056874:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757265606801 time 757313243235 state 0x0 ref 1 00000100:00100000:3.0:1789760768.056894:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757313503651 time 757313243235 state 0x1 ref 1 00000100:00100000:3.0:1789760768.056936:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757313503651 time 757387866397 state 0x0 ref 1 00000100:00100000:3.0:1789760768.056954:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484303360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760768.057010:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23531520, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:3.0:1789760768.057024:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.057041:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:3.0:1789760768.057045:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760768.057058:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23531520->23535615] (req 23531520->23535615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.057114:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23531520->23535615] (req 23531520->23535615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760768.057731:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:3.0:1789760768.105019:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23531520 -> 23535616 00010000:00010000:3.0:1789760768.105037:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23531520->23535615] (req 23531520->23535615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.105051:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23531520->23535615] (req 23531520->23535615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.105060:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23531520->23535615] (req 23531520->23535615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.105108:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760768.105112:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23531520->23535615] (req 23531520->23535615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.105131:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23531520->23535615] (req 23531520->23535615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.105151:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23531520->23535615] (req 23531520->23535615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.105159:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf38263962096d4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23531520->23535615] (req 23531520->23535615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760768.105239:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484303360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48289us (48558us total) trans 4294973063 rc 0/0 00000100:00100000:3.0:1789760768.105248:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5774 00000100:00100000:3.0:1789760768.105258:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757313503651 time 757387866397 state 0x2 ref 0 00000001:00080000:3.0:1789760768.105299:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973063 is committed 00000100:00100000:3.0:1789760768.122392:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760768.122566:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484303488 00000100:00100000:1.0:1789760768.122575:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484303488 00000100:00100000:1.0:1789760768.122624:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757313503651 time 757387866397 state 0x0 ref 1 00000100:00100000:1.0:1789760768.122708:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757388126813 time 757387866397 state 0x1 ref 1 00000100:00100000:1.0:1789760768.123242:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757388126813 time 757454171823 state 0x0 ref 1 00000100:00100000:1.0:1789760768.123256:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484303488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760768.123307:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23535616, mtime: 0x6aad94ff, atime: 0x0, ctime: 0x6aad94ff, blocks: 0xb807 00010000:00010000:1.0:1789760768.123317:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.123333:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:1.0:1789760768.123337:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760768.123344:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23535616->23539711] (req 23535616->23539711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.123358:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23535616->23539711] (req 23535616->23539711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760768.170344:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973064 is committed 00002000:00010000:2.0:1789760768.170434:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23535616 -> 23539712 00002000:00010000:2.0:1789760768.170439:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760767 -> 1789760768 00002000:00010000:2.0:1789760768.170441:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760767 -> 1789760768 00010000:00010000:2.0:1789760768.170454:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23535616->23539711] (req 23535616->23539711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.170466:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23535616->23539711] (req 23535616->23539711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.170474:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23535616->23539711] (req 23535616->23539711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.170481:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760768.170486:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23535616->23539711] (req 23535616->23539711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.170494:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23535616->23539711] (req 23535616->23539711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.170513:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23535616->23539711] (req 23535616->23539711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.170521:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962096db lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23535616->23539711] (req 23535616->23539711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760768.170608:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484303488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47352us (48221us total) trans 4294973064 rc 0/0 00000100:00100000:2.0:1789760768.170618:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5775 00000100:00100000:2.0:1789760768.170628:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757388126813 time 757454171823 state 0x2 ref 0 00000100:00100000:0.0:1789760768.175531:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760768.175943:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484303616 00000100:00100000:1.0:1789760768.175953:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484303616 00000100:00100000:1.0:1789760768.176005:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757388126813 time 757454171823 state 0x0 ref 1 00000100:00100000:1.0:1789760768.176019:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757454432239 time 757454171823 state 0x1 ref 1 00000100:00100000:1.0:1789760768.176049:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757454432239 time 757506980625 state 0x0 ref 1 00000100:00100000:1.0:1789760768.176062:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484303616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760768.176150:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23539712, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:1.0:1789760768.176159:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.176173:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:1.0:1789760768.176177:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760768.176184:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23539712->23543807] (req 23539712->23543807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.176197:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23539712->23543807] (req 23539712->23543807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760768.226803:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973065 is committed 00002000:00010000:0.0:1789760768.226973:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23539712 -> 23543808 00010000:00010000:0.0:1789760768.226990:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23539712->23543807] (req 23539712->23543807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.227002:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23539712->23543807] (req 23539712->23543807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.227011:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23539712->23543807] (req 23539712->23543807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.227018:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760768.227023:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23539712->23543807] (req 23539712->23543807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.227038:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23539712->23543807] (req 23539712->23543807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.227058:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23539712->23543807] (req 23539712->23543807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.227085:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf38263962096e2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23539712->23543807] (req 23539712->23543807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760768.227194:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484303616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51129us (51666us total) trans 4294973065 rc 0/0 00000100:00100000:3.0:1789760768.229169:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760768.229288:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484303744 00000100:00100000:1.0:1789760768.229298:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484303744 00000100:00100000:1.0:1789760768.229355:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 489 00000100:00100000:1.0:1789760768.229365:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c803a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484303744:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760768.229439:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c803a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484303744:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 76us (278us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760768.229447:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 489 00000100:00100000:3.0:1789760768.229840:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.229877:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.229914:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.229936:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.229957:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484304000 00000100:00100000:3.0:1789760768.229961:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484304000 00000100:00100000:3.0:1789760768.229999:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484304256 00000100:00100000:3.0:1789760768.230002:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484304256 00000100:00100000:3.0:1789760768.230021:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484303872 00000100:00100000:3.0:1789760768.230024:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484303872 00000100:00100000:3.0:1789760768.230036:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484304128 00000100:00100000:3.0:1789760768.230039:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484304128 00000100:00100000:3.0:1789760768.230052:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 971 00000100:00100000:3.0:1789760768.230058:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484304000:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760768.230138:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484304000:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 79us (298us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760768.230145:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 971 00000100:00100000:3.0:1789760768.230159:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 972 00000100:00100000:3.0:1789760768.230165:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484304256:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760768.230188:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484304256:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 23us (311us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760768.230193:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 972 00000100:00100000:3.0:1789760768.230203:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 973 00000100:00100000:3.0:1789760768.230753:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484303872:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760768.230783:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484303872:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 39us (869us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760768.230789:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 973 00000100:00100000:3.0:1789760768.230803:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 974 00000100:00100000:3.0:1789760768.230809:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abb480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484304128:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760768.230825:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abb480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484304128:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 17us (889us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760768.230831:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 974 00000100:00080000:3.0:1789760768.230855:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:3.0:1789760768.230859:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:0.0:1789760768.232098:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5776 00000100:00100000:0.0:1789760768.232112:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757454432239 time 757506980625 state 0x2 ref 0 00000100:00100000:3.0:1789760768.233741:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760768.233790:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484304384 00000100:00100000:0.0:1789760768.233799:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484304384 00000100:00100000:0.0:1789760768.233858:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757454432239 time 757506980625 state 0x0 ref 1 00000100:00100000:0.0:1789760768.233874:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757507241041 time 757506980625 state 0x1 ref 1 00000100:00100000:0.0:1789760768.233905:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757507241041 time 757564830651 state 0x0 ref 1 00000100:00100000:0.0:1789760768.233917:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484304384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760768.233964:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23543808, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:0.0:1789760768.233973:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.233988:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:0.0:1789760768.233992:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760768.233998:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23543808->23547903] (req 23543808->23547903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.234011:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23543808->23547903] (req 23543808->23547903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760768.279742:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973066 is committed 00002000:00010000:0.0:1789760768.279869:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23543808 -> 23547904 00010000:00010000:0.0:1789760768.279885:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23543808->23547903] (req 23543808->23547903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.279898:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23543808->23547903] (req 23543808->23547903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.279907:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23543808->23547903] (req 23543808->23547903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.279915:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760768.279919:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23543808->23547903] (req 23543808->23547903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.279927:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23543808->23547903] (req 23543808->23547903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.279948:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23543808->23547903] (req 23543808->23547903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.279957:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962096e9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23543808->23547903] (req 23543808->23547903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760768.280056:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484304384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46136us (46317us total) trans 4294973066 rc 0/0 00000100:00100000:0.0:1789760768.280107:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5777 00000100:00100000:0.0:1789760768.280127:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757507241041 time 757564830651 state 0x2 ref 0 00000100:00100000:1.0:1789760768.283974:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.284035:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484304512 00000100:00100000:3.0:1789760768.284046:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484304512 00000100:00100000:3.0:1789760768.284132:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757507241041 time 757564830651 state 0x0 ref 1 00000100:00100000:3.0:1789760768.284148:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757565091067 time 757564830651 state 0x1 ref 1 00000100:00100000:3.0:1789760768.284176:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757565091067 time 757615107450 state 0x0 ref 1 00000100:00100000:3.0:1789760768.284189:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484304512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760768.284245:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23547904, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:3.0:1789760768.284254:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.284270:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:3.0:1789760768.284274:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760768.284280:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23547904->23551999] (req 23547904->23551999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.284294:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23547904->23551999] (req 23547904->23551999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760768.342382:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973067 is committed 00002000:00010000:0.0:1789760768.342712:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23547904 -> 23552000 00010000:00010000:0.0:1789760768.342729:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23547904->23551999] (req 23547904->23551999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.342742:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23547904->23551999] (req 23547904->23551999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.342751:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23547904->23551999] (req 23547904->23551999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.342758:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760768.342763:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23547904->23551999] (req 23547904->23551999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.342772:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23547904->23551999] (req 23547904->23551999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.342796:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23547904->23551999] (req 23547904->23551999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.342804:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962096f0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23547904->23551999] (req 23547904->23551999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760768.342897:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484304512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58706us (58927us total) trans 4294973067 rc 0/0 00000100:00100000:0.0:1789760768.342907:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5778 00000100:00100000:0.0:1789760768.342916:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757565091067 time 757615107450 state 0x2 ref 0 00000100:00100000:3.0:1789760768.346705:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760768.347197:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484304640 00000100:00100000:2.0:1789760768.347207:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484304640 00000100:00100000:2.0:1789760768.347258:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757565091067 time 757615107450 state 0x0 ref 1 00000100:00100000:2.0:1789760768.347273:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757615367866 time 757615107450 state 0x1 ref 1 00000100:00100000:2.0:1789760768.348322:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757615367866 time 757679243827 state 0x0 ref 1 00000100:00100000:3.0:1789760768.348454:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484304640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760768.348511:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23552000, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:3.0:1789760768.348520:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.348537:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:3.0:1789760768.348541:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760768.348549:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23552000->23556095] (req 23552000->23556095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.348573:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23552000->23556095] (req 23552000->23556095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760768.391538:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973068 is committed 00002000:00010000:1.0:1789760768.391587:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23552000 -> 23556096 00010000:00010000:1.0:1789760768.391602:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23552000->23556095] (req 23552000->23556095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.391614:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23552000->23556095] (req 23552000->23556095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.391627:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23552000->23556095] (req 23552000->23556095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.391634:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760768.391638:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23552000->23556095] (req 23552000->23556095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.391646:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23552000->23556095] (req 23552000->23556095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.391665:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23552000->23556095] (req 23552000->23556095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.391672:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962096f7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23552000->23556095] (req 23552000->23556095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760768.391761:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484304640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43417us (45061us total) trans 4294973068 rc 0/0 00000100:00100000:1.0:1789760768.391771:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5779 00000100:00100000:1.0:1789760768.391780:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757615367866 time 757679243827 state 0x2 ref 0 00000100:00100000:3.0:1789760768.396679:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760768.396841:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484304768 00000100:00100000:0.0:1789760768.396851:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484304768 00000100:00100000:0.0:1789760768.396909:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757615367866 time 757679243827 state 0x0 ref 1 00000100:00100000:0.0:1789760768.396923:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757679504243 time 757679243827 state 0x1 ref 1 00000100:00100000:0.0:1789760768.396952:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757679504243 time 757727882984 state 0x0 ref 1 00000100:00100000:0.0:1789760768.396965:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abb800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484304768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760768.397011:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23556096, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:0.0:1789760768.397019:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.397034:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:0.0:1789760768.397039:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760768.397045:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23556096->23560191] (req 23556096->23560191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.397059:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23556096->23560191] (req 23556096->23560191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760768.434404:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973069 is committed 00002000:00010000:1.0:1789760768.434516:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23556096 -> 23560192 00010000:00010000:1.0:1789760768.434531:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23556096->23560191] (req 23556096->23560191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.434550:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23556096->23560191] (req 23556096->23560191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.434559:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23556096->23560191] (req 23556096->23560191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.434568:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760768.434573:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23556096->23560191] (req 23556096->23560191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.434583:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23556096->23560191] (req 23556096->23560191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.434603:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23556096->23560191] (req 23556096->23560191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.434612:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962096fe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23556096->23560191] (req 23556096->23560191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760768.434705:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abb800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484304768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37738us (38029us total) trans 4294973069 rc 0/0 00000100:00100000:1.0:1789760768.434717:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5780 00000100:00100000:1.0:1789760768.434726:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757679504243 time 757727882984 state 0x2 ref 0 00000100:00100000:0.0:1789760768.438568:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760768.438670:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484304896 00000100:00100000:2.0:1789760768.438680:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484304896 00000100:00100000:2.0:1789760768.438733:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757679504243 time 757727882984 state 0x0 ref 1 00000100:00100000:2.0:1789760768.438749:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757728143400 time 757727882984 state 0x1 ref 1 00000100:00100000:2.0:1789760768.438819:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757728143400 time 757769748372 state 0x0 ref 1 00000100:00100000:2.0:1789760768.438832:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484304896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760768.438881:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23560192, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:2.0:1789760768.438891:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.438919:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:2.0:1789760768.438924:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760768.438930:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23560192->23564287] (req 23560192->23564287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.438956:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23560192->23564287] (req 23560192->23564287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760768.489439:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23560192 -> 23564288 00010000:00010000:0.0:1789760768.489459:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23560192->23564287] (req 23560192->23564287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.489473:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23560192->23564287] (req 23560192->23564287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.489483:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23560192->23564287] (req 23560192->23564287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.489490:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760768.489495:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23560192->23564287] (req 23560192->23564287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.489505:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23560192->23564287] (req 23560192->23564287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.489525:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23560192->23564287] (req 23560192->23564287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.489537:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209705 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23560192->23564287] (req 23560192->23564287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760768.489627:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484304896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50794us (51065us total) trans 4294973070 rc 0/0 00000100:00100000:0.0:1789760768.489638:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5781 00000100:00100000:0.0:1789760768.489648:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757728143400 time 757769748372 state 0x2 ref 0 00000001:00080000:1.0:1789760768.490332:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973070 is committed 00000100:00100000:3.0:1789760768.493738:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760768.493826:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484305024 00000100:00100000:0.0:1789760768.493835:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484305024 00000100:00100000:0.0:1789760768.493893:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757728143400 time 757769748372 state 0x0 ref 1 00000100:00100000:0.0:1789760768.493964:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757770008788 time 757769748372 state 0x1 ref 1 00000100:00100000:0.0:1789760768.494053:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757770008788 time 757824983105 state 0x0 ref 1 00000100:00100000:0.0:1789760768.494099:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706e450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484305024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760768.494159:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23564288, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:0.0:1789760768.494168:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.494183:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:0.0:1789760768.494187:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760768.494193:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23564288->23568383] (req 23564288->23568383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.494212:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23564288->23568383] (req 23564288->23568383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760768.516973:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973071 is committed 00002000:00010000:0.0:1789760768.517060:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23564288 -> 23568384 00010000:00010000:0.0:1789760768.517095:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23564288->23568383] (req 23564288->23568383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.517109:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23564288->23568383] (req 23564288->23568383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.517130:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23564288->23568383] (req 23564288->23568383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.517138:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760768.517143:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23564288->23568383] (req 23564288->23568383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.517152:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23564288->23568383] (req 23564288->23568383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.517171:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23564288->23568383] (req 23564288->23568383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.517178:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620970c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23564288->23568383] (req 23564288->23568383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760768.517274:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706e450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484305024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23205us (23540us total) trans 4294973071 rc 0/0 00000100:00100000:0.0:1789760768.517284:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5782 00000100:00100000:0.0:1789760768.517329:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757770008788 time 757824983105 state 0x2 ref 0 00000100:00100000:3.0:1789760768.520460:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760768.520765:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484305152 00000100:00100000:1.0:1789760768.520775:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484305152 00000100:00100000:1.0:1789760768.520829:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757770008788 time 757824983105 state 0x0 ref 1 00000100:00100000:1.0:1789760768.520846:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757825243521 time 757824983105 state 0x1 ref 1 00000100:00100000:1.0:1789760768.520878:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757825243521 time 757851809539 state 0x0 ref 1 00000100:00100000:1.0:1789760768.520890:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484305152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760768.521003:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23568384, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:1.0:1789760768.521013:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.521028:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:1.0:1789760768.521032:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760768.521038:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23568384->23572479] (req 23568384->23572479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.521051:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23568384->23572479] (req 23568384->23572479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760768.559847:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973072 is committed 00002000:00010000:1.0:1789760768.559928:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23568384 -> 23572480 00010000:00010000:1.0:1789760768.559941:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23568384->23572479] (req 23568384->23572479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.559950:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23568384->23572479] (req 23568384->23572479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.559957:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23568384->23572479] (req 23568384->23572479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.559962:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760768.559965:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23568384->23572479] (req 23568384->23572479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.559971:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23568384->23572479] (req 23568384->23572479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.559985:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23568384->23572479] (req 23568384->23572479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.559990:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209713 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23568384->23572479] (req 23568384->23572479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760768.560093:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484305152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39173us (39612us total) trans 4294973072 rc 0/0 00000100:00100000:1.0:1789760768.560100:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5783 00000100:00100000:1.0:1789760768.560108:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757825243521 time 757851809539 state 0x2 ref 0 00000100:00100000:1.0:1789760768.562802:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760768.563175:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484305280 00000100:00100000:0.0:1789760768.563184:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484305280 00000100:00100000:0.0:1789760768.563295:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757825243521 time 757851809539 state 0x0 ref 1 00000100:00100000:0.0:1789760768.563309:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757852069955 time 757851809539 state 0x1 ref 1 00000100:00100000:0.0:1789760768.563371:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757852069955 time 757894302204 state 0x0 ref 1 00000100:00100000:0.0:1789760768.563385:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484305280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760768.563430:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23572480, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:0.0:1789760768.563439:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.563454:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:0.0:1789760768.563458:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760768.563464:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23572480->23576575] (req 23572480->23576575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.563478:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23572480->23576575] (req 23572480->23576575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760768.588453:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973073 is committed 00002000:00010000:3.0:1789760768.588710:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23572480 -> 23576576 00010000:00010000:3.0:1789760768.588723:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23572480->23576575] (req 23572480->23576575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.588735:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23572480->23576575] (req 23572480->23576575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.588744:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23572480->23576575] (req 23572480->23576575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.588752:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760768.588756:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23572480->23576575] (req 23572480->23576575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.588766:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23572480->23576575] (req 23572480->23576575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.588791:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23572480->23576575] (req 23572480->23576575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.588800:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620971a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23572480->23576575] (req 23572480->23576575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760768.588943:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484305280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25556us (26144us total) trans 4294973073 rc 0/0 00000100:00100000:3.0:1789760768.588954:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5784 00000100:00100000:3.0:1789760768.588964:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757852069955 time 757894302204 state 0x2 ref 0 00000100:00100000:3.0:1789760768.593604:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760768.594037:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484305408 00000100:00100000:0.0:1789760768.594048:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484305408 00000100:00100000:0.0:1789760768.594160:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757852069955 time 757894302204 state 0x0 ref 1 00000100:00100000:0.0:1789760768.594175:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757894562620 time 757894302204 state 0x1 ref 1 00000100:00100000:0.0:1789760768.594195:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757894562620 time 757925126477 state 0x0 ref 1 00000100:00100000:0.0:1789760768.594220:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484305408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760768.594268:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23576576, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:0.0:1789760768.594277:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.594291:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:0.0:1789760768.594296:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760768.594302:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23576576->23580671] (req 23576576->23580671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.594317:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23576576->23580671] (req 23576576->23580671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760768.632348:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973074 is committed 00002000:00010000:1.0:1789760768.632477:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23576576 -> 23580672 00010000:00010000:1.0:1789760768.632492:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23576576->23580671] (req 23576576->23580671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.632505:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23576576->23580671] (req 23576576->23580671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.632514:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23576576->23580671] (req 23576576->23580671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.632520:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760768.632525:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23576576->23580671] (req 23576576->23580671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.632532:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23576576->23580671] (req 23576576->23580671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.632549:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23576576->23580671] (req 23576576->23580671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.632556:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209721 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23576576->23580671] (req 23576576->23580671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760768.632649:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484305408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38428us (39050us total) trans 4294973074 rc 0/0 00000100:00100000:1.0:1789760768.632659:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5785 00000100:00100000:1.0:1789760768.632673:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757894562620 time 757925126477 state 0x2 ref 0 00000100:00100000:0.0:1789760768.637467:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.637602:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484305536 00000100:00100000:3.0:1789760768.637612:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484305536 00000100:00100000:3.0:1789760768.637667:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757894562620 time 757925126477 state 0x0 ref 1 00000100:00100000:3.0:1789760768.637682:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757925386893 time 757925126477 state 0x1 ref 1 00000100:00100000:3.0:1789760768.637713:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757925386893 time 757968644428 state 0x0 ref 1 00000100:00100000:3.0:1789760768.637726:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484305536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760768.637774:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23580672, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:3.0:1789760768.637784:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.637799:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=757s v=(0 0) 00010000:00010000:3.0:1789760768.637804:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760768.637810:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23580672->23584767] (req 23580672->23584767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.637824:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23580672->23584767] (req 23580672->23584767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760768.682836:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23580672 -> 23584768 00010000:00010000:0.0:1789760768.682857:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23580672->23584767] (req 23580672->23584767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.682874:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23580672->23584767] (req 23580672->23584767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.682884:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23580672->23584767] (req 23580672->23584767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.682891:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760768.682905:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23580672->23584767] (req 23580672->23584767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.683008:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23580672->23584767] (req 23580672->23584767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.683030:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23580672->23584767] (req 23580672->23584767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.683038:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209728 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23580672->23584767] (req 23580672->23584767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760768.683169:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484305536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45440us (45716us total) trans 4294973075 rc 0/0 00000100:00100000:0.0:1789760768.683180:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5786 00000100:00100000:0.0:1789760768.683189:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757925386893 time 757968644428 state 0x2 ref 0 00000001:00080000:1.0:1789760768.683775:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973075 is committed 00000100:00100000:3.0:1789760768.687899:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760768.688275:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484305664 00000100:00100000:1.0:1789760768.688286:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484305664 00000100:00100000:1.0:1789760768.688343:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757925386893 time 757968644428 state 0x0 ref 1 00000100:00100000:1.0:1789760768.688359:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757968904844 time 757968644428 state 0x1 ref 1 00000100:00100000:1.0:1789760768.688426:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757968904844 time 758019357069 state 0x0 ref 1 00000100:00100000:1.0:1789760768.688444:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484305664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760768.688501:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23584768, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:1.0:1789760768.688510:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.688526:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:1.0:1789760768.688530:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760768.688536:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23584768->23588863] (req 23584768->23588863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.688549:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23584768->23588863] (req 23584768->23588863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760768.723092:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973076 is committed 00002000:00010000:3.0:1789760768.723279:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23584768 -> 23588864 00010000:00010000:3.0:1789760768.723292:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23584768->23588863] (req 23584768->23588863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.723310:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23584768->23588863] (req 23584768->23588863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.723316:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23584768->23588863] (req 23584768->23588863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.723321:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760768.723325:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23584768->23588863] (req 23584768->23588863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.723330:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23584768->23588863] (req 23584768->23588863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.723346:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23584768->23588863] (req 23584768->23588863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.723354:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620972f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23584768->23588863] (req 23584768->23588863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760768.723446:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484305664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35005us (35553us total) trans 4294973076 rc 0/0 00000100:00100000:3.0:1789760768.723728:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5787 00000100:00100000:3.0:1789760768.723738:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757968904844 time 758019357069 state 0x2 ref 0 00000100:00100000:0.0:1789760768.729570:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.731747:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484305792 00000100:00100000:3.0:1789760768.731761:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484305792 00000100:00100000:2.0:1789760768.731907:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 757968904844 time 758019357069 state 0x0 ref 1 00000100:00100000:2.0:1789760768.731924:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758019617485 time 758019357069 state 0x1 ref 1 00000100:00100000:2.0:1789760768.731996:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758019617485 time 758062926344 state 0x0 ref 1 00000100:00100000:2.0:1789760768.732010:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484305792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760768.732059:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23588864, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:2.0:1789760768.732098:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.732125:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:2.0:1789760768.732130:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760768.732136:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23588864->23592959] (req 23588864->23592959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.732152:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23588864->23592959] (req 23588864->23592959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760768.778345:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23588864 -> 23592960 00010000:00010000:0.0:1789760768.778366:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23588864->23592959] (req 23588864->23592959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.778380:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23588864->23592959] (req 23588864->23592959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.778390:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23588864->23592959] (req 23588864->23592959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.778398:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760768.778403:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23588864->23592959] (req 23588864->23592959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.778412:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23588864->23592959] (req 23588864->23592959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.778431:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23588864->23592959] (req 23588864->23592959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.778440:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209736 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23588864->23592959] (req 23588864->23592959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760768.778540:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484305792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46528us (48975us total) trans 4294973077 rc 0/0 00000100:00100000:0.0:1789760768.778550:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5788 00000100:00100000:0.0:1789760768.778563:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758019617485 time 758062926344 state 0x2 ref 0 00000001:00080000:3.0:1789760768.778883:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973077 is committed 00000100:00100000:0.0:1789760768.784875:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760768.785492:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484305920 00000100:00100000:1.0:1789760768.785502:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484305920 00000100:00100000:1.0:1789760768.785552:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758019617485 time 758062926344 state 0x0 ref 1 00000100:00100000:1.0:1789760768.785566:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758063186760 time 758062926344 state 0x1 ref 1 00000100:00100000:1.0:1789760768.785646:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758063186760 time 758116575849 state 0x0 ref 1 00000100:00100000:1.0:1789760768.785659:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484305920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760768.785708:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23592960, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:1.0:1789760768.785726:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.785742:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:1.0:1789760768.785747:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760768.785754:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23592960->23597055] (req 23592960->23597055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.785768:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23592960->23597055] (req 23592960->23597055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760768.826832:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973078 is committed 00002000:00010000:1.0:1789760768.826988:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23592960 -> 23597056 00010000:00010000:1.0:1789760768.827003:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23592960->23597055] (req 23592960->23597055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.827017:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23592960->23597055] (req 23592960->23597055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.827026:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23592960->23597055] (req 23592960->23597055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.827033:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760768.827037:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23592960->23597055] (req 23592960->23597055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.827046:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23592960->23597055] (req 23592960->23597055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.827146:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23592960->23597055] (req 23592960->23597055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.827155:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620973d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23592960->23597055] (req 23592960->23597055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760768.827404:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484305920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41742us (42532us total) trans 4294973078 rc 0/0 00000100:00100000:1.0:1789760768.827479:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5789 00000100:00100000:1.0:1789760768.827490:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758063186760 time 758116575849 state 0x2 ref 0 00000100:00100000:0.0:1789760768.831728:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.831823:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484306048 00000100:00100000:3.0:1789760768.831833:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484306048 00000100:00100000:3.0:1789760768.831885:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758063186760 time 758116575849 state 0x0 ref 1 00000100:00100000:3.0:1789760768.831963:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758116836265 time 758116575849 state 0x1 ref 1 00000100:00100000:3.0:1789760768.832030:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758116836265 time 758162960543 state 0x0 ref 1 00000100:00100000:3.0:1789760768.832042:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484306048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760768.832142:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23597056, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:3.0:1789760768.832151:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.832168:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:3.0:1789760768.832172:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760768.832179:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23597056->23601151] (req 23597056->23601151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.832207:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23597056->23601151] (req 23597056->23601151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760768.866820:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973079 is committed 00002000:00010000:0.0:1789760768.866981:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23597056 -> 23601152 00010000:00010000:0.0:1789760768.866997:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23597056->23601151] (req 23597056->23601151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.867009:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23597056->23601151] (req 23597056->23601151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.867018:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23597056->23601151] (req 23597056->23601151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.867027:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760768.867031:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23597056->23601151] (req 23597056->23601151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.867040:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23597056->23601151] (req 23597056->23601151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.867058:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23597056->23601151] (req 23597056->23601151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.867156:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209744 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23597056->23601151] (req 23597056->23601151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760768.867543:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484306048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35496us (35818us total) trans 4294973079 rc 0/0 00000100:00100000:2.0:1789760768.867555:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5790 00000100:00100000:2.0:1789760768.867565:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758116836265 time 758162960543 state 0x2 ref 0 00000100:00100000:1.0:1789760768.871733:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760768.871828:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484306176 00000100:00100000:3.0:1789760768.871839:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484306176 00000100:00100000:3.0:1789760768.871891:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758116836265 time 758162960543 state 0x0 ref 1 00000100:00100000:3.0:1789760768.871906:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758163220959 time 758162960543 state 0x1 ref 1 00000100:00100000:3.0:1789760768.871936:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758163220959 time 758202867265 state 0x0 ref 1 00000100:00100000:3.0:1789760768.871949:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484306176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760768.871999:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23601152, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:3.0:1789760768.872131:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.872148:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:3.0:1789760768.872152:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760768.872159:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23601152->23605247] (req 23601152->23605247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760768.872347:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23601152->23605247] (req 23601152->23605247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760768.900373:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973080 is committed 00002000:00010000:1.0:1789760768.900526:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23601152 -> 23605248 00010000:00010000:1.0:1789760768.900541:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23601152->23605247] (req 23601152->23605247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.900553:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23601152->23605247] (req 23601152->23605247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.900563:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23601152->23605247] (req 23601152->23605247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.900575:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760768.900579:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23601152->23605247] (req 23601152->23605247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.900590:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23601152->23605247] (req 23601152->23605247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.900609:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23601152->23605247] (req 23601152->23605247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760768.900617:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620974b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23601152->23605247] (req 23601152->23605247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760768.900805:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484306176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28853us (29075us total) trans 4294973080 rc 0/0 00000100:00100000:1.0:1789760768.900816:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5791 00000100:00100000:1.0:1789760768.900826:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758163220959 time 758202867265 state 0x2 ref 0 00000100:00100000:0.0:1789760768.906030:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00010000:00010000:3.0:1789760768.908243:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760768.908547:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8ab8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508933888:0@lo:13:osp-pre-0-0.0 00000100:00100000:1.0:1789760768.908612:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760768.909269:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508933888 00000100:00100000:3.0:1789760768.909277:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508933888 00000100:00100000:3.0:1789760768.909326:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 402 00000100:00100000:3.0:1789760768.909336:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508933888:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:3.0:1789760768.909359:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 526057472 pending 0 free 3834642432 avail 3807379456 00000100:00100000:3.0:1789760768.909422:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508933888:12345-0@lo:13:osp-pre-0-0.0 Request processed in 82us (803us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760768.909429:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 402 00000100:00100000:0.0:1789760768.909523:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8ab8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508933888:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1789760768.911748:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484306304 00000100:00100000:2.0:1789760768.911765:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484306304 00000100:00100000:2.0:1789760768.911822:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758163220959 time 758202867265 state 0x0 ref 1 00000100:00100000:0.0:1789760768.911837:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760768.911838:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758203127681 time 758202867265 state 0x1 ref 1 00000100:00100000:0.0:1789760768.911851:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4defddf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508934016:0@lo:41:osp-pre-0-1.0 00000100:00100000:2.0:1789760768.911858:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758203127681 time 758242790142 state 0x0 ref 1 00000100:00100000:2.0:1789760768.911868:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484306304:12345-192.168.203.10@tcp:4:dd.0 00000100:00100000:0.0:1789760768.911907:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00002000:00010000:2.0:1789760768.911911:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23605248, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:2.0:1789760768.911920:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.911934:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:2.0:1789760768.911938:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760768.911945:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23605248->23609343] (req 23605248->23609343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.911969:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23605248->23609343] (req 23605248->23609343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760768.912339:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508934144:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760768.912368:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760768.912431:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508934016 00000100:00100000:1.0:1789760768.912436:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508934016 00000100:00100000:1.0:1789760768.912488:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 461 00000100:00100000:1.0:1789760768.912497:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508934016:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:1.0:1789760768.912534:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:1.0:1789760768.912581:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508934016:12345-0@lo:41:osp-pre-0-1.0 Request processed in 86us (672us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760768.912589:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 461 00000100:00100000:1.0:1789760768.912611:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508934144 00000100:00100000:1.0:1789760768.912614:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508934144 00000100:00100000:1.0:1789760768.912636:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 403 00000100:00100000:1.0:1789760768.912642:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508934144:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:1.0:1789760768.912655:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528154624 pending 1703936 free 3834642432 avail 3807379456 00000100:00100000:1.0:1789760768.913837:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508934144:12345-0@lo:13:osp-pre-0-1.0 Request processed in 1189us (1462us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760768.913848:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 403 00000100:00100000:1.0:1789760768.913905:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4defddf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508934016:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760768.916054:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508934144:0@lo:13:osp-pre-0-1.0 00000001:00080000:1.0:1789760768.962511:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973081 is committed 00002000:00010000:2.0:1789760768.962618:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23605248 -> 23609344 00010000:00010000:2.0:1789760768.962633:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23605248->23609343] (req 23605248->23609343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.962644:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23605248->23609343] (req 23605248->23609343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.962651:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23605248->23609343] (req 23605248->23609343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.962659:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760768.962663:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23605248->23609343] (req 23605248->23609343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.962671:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23605248->23609343] (req 23605248->23609343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.962689:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23605248->23609343] (req 23605248->23609343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760768.962696:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209752 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23605248->23609343] (req 23605248->23609343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760768.962791:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484306304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50919us (56766us total) trans 4294973081 rc 0/0 00000100:00100000:2.0:1789760768.962802:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5792 00000100:00100000:2.0:1789760768.962813:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758203127681 time 758242790142 state 0x2 ref 0 00000100:00100000:1.0:1789760768.967423:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760768.969456:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484306432 00000100:00100000:0.0:1789760768.969471:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484306432 00000100:00100000:0.0:1789760768.969536:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758203127681 time 758242790142 state 0x0 ref 1 00000100:00100000:0.0:1789760768.969552:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758243050558 time 758242790142 state 0x1 ref 1 00000100:00100000:0.0:1789760768.969589:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758243050558 time 758300519776 state 0x0 ref 1 00000100:00100000:0.0:1789760768.969604:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484306432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760768.969652:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23609344, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:0.0:1789760768.969661:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.969677:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:0.0:1789760768.969682:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760768.969689:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23609344->23613439] (req 23609344->23613439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760768.969703:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23609344->23613439] (req 23609344->23613439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760769.011594:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973082 is committed 00002000:00010000:2.0:1789760769.012411:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23609344 -> 23613440 00010000:00010000:2.0:1789760769.012427:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23609344->23613439] (req 23609344->23613439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.012439:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23609344->23613439] (req 23609344->23613439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.012448:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23609344->23613439] (req 23609344->23613439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.012455:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760769.012459:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23609344->23613439] (req 23609344->23613439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.012469:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23609344->23613439] (req 23609344->23613439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.012489:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23609344->23613439] (req 23609344->23613439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.012501:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209759 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23609344->23613439] (req 23609344->23613439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760769.012602:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484306432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42997us (45184us total) trans 4294973082 rc 0/0 00000100:00100000:2.0:1789760769.012613:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5793 00000100:00100000:2.0:1789760769.012623:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758243050558 time 758300519776 state 0x2 ref 0 00000100:00100000:1.0:1789760769.016169:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760769.016358:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484306560 00000100:00100000:3.0:1789760769.016369:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484306560 00000100:00100000:3.0:1789760769.016424:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758243050558 time 758300519776 state 0x0 ref 1 00000100:00100000:3.0:1789760769.016439:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758300780192 time 758300519776 state 0x1 ref 1 00000100:00100000:3.0:1789760769.016467:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758300780192 time 758347398693 state 0x0 ref 1 00000100:00100000:3.0:1789760769.016480:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484306560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760769.016527:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23613440, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:3.0:1789760769.016535:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.016550:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:3.0:1789760769.016555:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760769.016562:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23613440->23617535] (req 23613440->23617535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.016575:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23613440->23617535] (req 23613440->23617535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760769.048918:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973083 is committed 00002000:00010000:3.0:1789760769.049094:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23613440 -> 23617536 00010000:00010000:3.0:1789760769.049119:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23613440->23617535] (req 23613440->23617535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.049132:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23613440->23617535] (req 23613440->23617535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.049141:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23613440->23617535] (req 23613440->23617535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.049148:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760769.049153:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23613440->23617535] (req 23613440->23617535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.049162:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23613440->23617535] (req 23613440->23617535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.049181:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23613440->23617535] (req 23613440->23617535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.049189:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209760 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23613440->23617535] (req 23613440->23617535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760769.049384:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484306560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32901us (33219us total) trans 4294973083 rc 0/0 00000100:00100000:3.0:1789760769.049395:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5794 00000100:00100000:3.0:1789760769.049404:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758300780192 time 758347398693 state 0x2 ref 0 00000100:00100000:0.0:1789760769.053356:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760769.053462:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484306688 00000100:00100000:1.0:1789760769.053471:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484306688 00000100:00100000:1.0:1789760769.053525:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758300780192 time 758347398693 state 0x0 ref 1 00000100:00100000:1.0:1789760769.053539:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758347659109 time 758347398693 state 0x1 ref 1 00000100:00100000:1.0:1789760769.053604:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758347659109 time 758384534436 state 0x0 ref 1 00000100:00100000:1.0:1789760769.053617:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484306688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760769.053666:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23617536, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:1.0:1789760769.053675:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.053693:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:1.0:1789760769.053698:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760769.053705:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23617536->23621631] (req 23617536->23621631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.053718:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23617536->23621631] (req 23617536->23621631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760769.080890:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23617536 -> 23621632 00010000:00010000:1.0:1789760769.080921:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23617536->23621631] (req 23617536->23621631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.080934:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23617536->23621631] (req 23617536->23621631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.080944:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23617536->23621631] (req 23617536->23621631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.080952:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760769.080956:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23617536->23621631] (req 23617536->23621631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.080966:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23617536->23621631] (req 23617536->23621631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.080986:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23617536->23621631] (req 23617536->23621631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.080994:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209767 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23617536->23621631] (req 23617536->23621631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760769.081127:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484306688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27507us (27776us total) trans 4294973084 rc 0/0 00000100:00100000:1.0:1789760769.081139:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5795 00000100:00100000:1.0:1789760769.081149:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758347659109 time 758384534436 state 0x2 ref 0 00000001:00080000:3.0:1789760769.082476:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973084 is committed 00000100:00100000:1.0:1789760769.084435:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760769.084534:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484306816 00000100:00100000:2.0:1789760769.084545:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484306816 00000100:00100000:2.0:1789760769.084605:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758347659109 time 758384534436 state 0x0 ref 1 00000100:00100000:2.0:1789760769.084622:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758384794852 time 758384534436 state 0x1 ref 1 00000100:00100000:2.0:1789760769.084701:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758384794852 time 758415631375 state 0x0 ref 1 00000100:00100000:2.0:1789760769.084715:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484306816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760769.084766:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23621632, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:2.0:1789760769.084775:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.084792:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:2.0:1789760769.084798:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760769.084805:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23621632->23625727] (req 23621632->23625727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.084820:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23621632->23625727] (req 23621632->23625727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760769.115656:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973085 is committed 00002000:00010000:2.0:1789760769.115979:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23621632 -> 23625728 00010000:00010000:2.0:1789760769.115995:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23621632->23625727] (req 23621632->23625727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.116008:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23621632->23625727] (req 23621632->23625727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.116016:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23621632->23625727] (req 23621632->23625727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.116024:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760769.116028:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23621632->23625727] (req 23621632->23625727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.116265:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23621632->23625727] (req 23621632->23625727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.116289:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23621632->23625727] (req 23621632->23625727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.116295:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620976e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23621632->23625727] (req 23621632->23625727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760769.116427:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484306816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31709us (31997us total) trans 4294973085 rc 0/0 00000100:00100000:2.0:1789760769.116437:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5796 00000100:00100000:2.0:1789760769.116447:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758384794852 time 758415631375 state 0x2 ref 0 00000100:00100000:0.0:1789760769.120855:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760769.121006:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484306944 00000100:00100000:2.0:1789760769.121015:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484306944 00000100:00100000:2.0:1789760769.122160:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758384794852 time 758415631375 state 0x0 ref 1 00000100:00100000:2.0:1789760769.122244:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758415891791 time 758415631375 state 0x1 ref 1 00000100:00100000:2.0:1789760769.122281:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758415891791 time 758453212677 state 0x0 ref 1 00000100:00100000:2.0:1789760769.122297:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484306944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760769.122349:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23625728, mtime: 0x6aad9500, atime: 0x0, ctime: 0x6aad9500, blocks: 0xb807 00010000:00010000:2.0:1789760769.122357:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.122372:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:2.0:1789760769.122376:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760769.122381:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23625728->23629823] (req 23625728->23629823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.122393:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23625728->23629823] (req 23625728->23629823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760769.153961:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973086 is committed 00002000:00010000:3.0:1789760769.154133:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23625728 -> 23629824 00002000:00010000:3.0:1789760769.154138:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760768 -> 1789760769 00002000:00010000:3.0:1789760769.154141:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760768 -> 1789760769 00010000:00010000:3.0:1789760769.154153:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23625728->23629823] (req 23625728->23629823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.154166:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23625728->23629823] (req 23625728->23629823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.154175:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23625728->23629823] (req 23625728->23629823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.154182:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760769.154186:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23625728->23629823] (req 23625728->23629823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.154194:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23625728->23629823] (req 23625728->23629823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.154285:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23625728->23629823] (req 23625728->23629823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.154291:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf3826396209775 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23625728->23629823] (req 23625728->23629823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760769.154383:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484306944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32086us (33534us total) trans 4294973086 rc 0/0 00000100:00100000:3.0:1789760769.154393:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5797 00000100:00100000:3.0:1789760769.154403:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758415891791 time 758453212677 state 0x2 ref 0 00000100:00100000:0.0:1789760769.159667:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760769.159755:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484307072 00000100:00100000:3.0:1789760769.159765:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484307072 00000100:00100000:3.0:1789760769.159818:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758415891791 time 758453212677 state 0x0 ref 1 00000100:00100000:3.0:1789760769.159832:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758453473093 time 758453212677 state 0x1 ref 1 00000100:00100000:3.0:1789760769.159892:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758453473093 time 758490822305 state 0x0 ref 1 00000100:00100000:3.0:1789760769.159904:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484307072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760769.159953:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23629824, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:3.0:1789760769.159961:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.159977:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:3.0:1789760769.159982:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760769.159988:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23629824->23633919] (req 23629824->23633919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.160002:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23629824->23633919] (req 23629824->23633919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760769.195285:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973087 is committed 00002000:00010000:0.0:1789760769.196907:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23629824 -> 23633920 00010000:00010000:0.0:1789760769.196925:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23629824->23633919] (req 23629824->23633919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.196939:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23629824->23633919] (req 23629824->23633919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.196948:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23629824->23633919] (req 23629824->23633919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.196955:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760769.196960:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23629824->23633919] (req 23629824->23633919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.196969:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23629824->23633919] (req 23629824->23633919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.196988:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23629824->23633919] (req 23629824->23633919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.196996:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620977c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23629824->23633919] (req 23629824->23633919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760769.201212:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484307072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41298us (41542us total) trans 4294973087 rc 0/0 00000100:00100000:0.0:1789760769.201227:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5798 00000100:00100000:0.0:1789760769.201237:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758453473093 time 758490822305 state 0x2 ref 0 00000100:00100000:1.0:1789760769.211364:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760769.211650:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484307200 00000100:00100000:3.0:1789760769.211660:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484307200 00000100:00100000:3.0:1789760769.211713:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758453473093 time 758490822305 state 0x0 ref 1 00000100:00100000:3.0:1789760769.211727:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758491082721 time 758490822305 state 0x1 ref 1 00000100:00100000:3.0:1789760769.211761:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758491082721 time 758542691872 state 0x0 ref 1 00000100:00100000:3.0:1789760769.211773:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484307200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760769.211826:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23633920, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:3.0:1789760769.211836:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.211851:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:3.0:1789760769.211856:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760769.211862:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23633920->23638015] (req 23633920->23638015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.211876:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23633920->23638015] (req 23633920->23638015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760769.261371:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973088 is committed 00002000:00010000:0.0:1789760769.261479:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23633920 -> 23638016 00010000:00010000:0.0:1789760769.261495:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23633920->23638015] (req 23633920->23638015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.261507:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23633920->23638015] (req 23633920->23638015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.261516:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23633920->23638015] (req 23633920->23638015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.261524:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760769.261529:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23633920->23638015] (req 23633920->23638015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.261538:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23633920->23638015] (req 23633920->23638015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.261558:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23633920->23638015] (req 23633920->23638015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.261570:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209783 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23633920->23638015] (req 23633920->23638015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760769.261673:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484307200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49888us (50305us total) trans 4294973088 rc 0/0 00000100:00100000:0.0:1789760769.261683:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5799 00000100:00100000:0.0:1789760769.261693:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758491082721 time 758542691872 state 0x2 ref 0 00000100:00100000:2.0:1789760769.268243:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760769.271574:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484307328 00000100:00100000:0.0:1789760769.271588:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484307328 00000100:00100000:0.0:1789760769.271657:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758491082721 time 758542691872 state 0x0 ref 1 00000100:00100000:0.0:1789760769.271673:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758542952288 time 758542691872 state 0x1 ref 1 00000100:00100000:0.0:1789760769.271748:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758542952288 time 758602678368 state 0x0 ref 1 00000100:00100000:0.0:1789760769.271764:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484307328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760769.271810:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23638016, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:0.0:1789760769.271820:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.271836:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:0.0:1789760769.271840:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760769.271846:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23638016->23642111] (req 23638016->23642111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.271865:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23638016->23642111] (req 23638016->23642111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760769.309705:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973089 is committed 00002000:00010000:2.0:1789760769.309898:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23638016 -> 23642112 00010000:00010000:2.0:1789760769.309913:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23638016->23642111] (req 23638016->23642111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.309926:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23638016->23642111] (req 23638016->23642111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.309935:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23638016->23642111] (req 23638016->23642111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.309943:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760769.309948:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23638016->23642111] (req 23638016->23642111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.309959:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23638016->23642111] (req 23638016->23642111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.309979:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23638016->23642111] (req 23638016->23642111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.309987:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620978a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23638016->23642111] (req 23638016->23642111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760769.310137:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484307328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38372us (41898us total) trans 4294973089 rc 0/0 00000100:00100000:2.0:1789760769.310147:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5800 00000100:00100000:2.0:1789760769.310157:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758542952288 time 758602678368 state 0x2 ref 0 00000100:00100000:2.0:1789760769.317093:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760769.317273:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484307456 00000100:00100000:3.0:1789760769.317283:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484307456 00000100:00100000:3.0:1789760769.317331:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758542952288 time 758602678368 state 0x0 ref 1 00000100:00100000:3.0:1789760769.317346:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758602938784 time 758602678368 state 0x1 ref 1 00000100:00100000:3.0:1789760769.317375:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758602938784 time 758648305996 state 0x0 ref 1 00000100:00100000:3.0:1789760769.317386:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484307456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760769.317430:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23642112, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:3.0:1789760769.317438:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.317454:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:3.0:1789760769.317458:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760769.317465:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23642112->23646207] (req 23642112->23646207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.317479:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23642112->23646207] (req 23642112->23646207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760769.365053:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973090 is committed 00002000:00010000:2.0:1789760769.365194:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23642112 -> 23646208 00010000:00010000:2.0:1789760769.365210:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23642112->23646207] (req 23642112->23646207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.365223:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23642112->23646207] (req 23642112->23646207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.365233:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23642112->23646207] (req 23642112->23646207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.365241:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760769.365246:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23642112->23646207] (req 23642112->23646207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.365255:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23642112->23646207] (req 23642112->23646207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.365276:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23642112->23646207] (req 23642112->23646207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.365289:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209791 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23642112->23646207] (req 23642112->23646207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760769.365376:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484307456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47986us (48288us total) trans 4294973090 rc 0/0 00000100:00100000:2.0:1789760769.365387:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5801 00000100:00100000:2.0:1789760769.365397:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758602938784 time 758648305996 state 0x2 ref 0 00000100:00100000:3.0:1789760769.368105:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760769.368213:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484307584 00000100:00100000:2.0:1789760769.368223:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484307584 00000100:00100000:2.0:1789760769.368280:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758602938784 time 758648305996 state 0x0 ref 1 00000100:00100000:2.0:1789760769.368297:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758648566412 time 758648305996 state 0x1 ref 1 00000100:00100000:2.0:1789760769.368328:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758648566412 time 758699259463 state 0x0 ref 1 00000100:00100000:2.0:1789760769.368343:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abb100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484307584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760769.368391:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23646208, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:2.0:1789760769.368400:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.368417:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:2.0:1789760769.368421:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760769.368427:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23646208->23650303] (req 23646208->23650303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.368444:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23646208->23650303] (req 23646208->23650303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760769.418338:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973091 is committed 00002000:00010000:2.0:1789760769.418412:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23646208 -> 23650304 00010000:00010000:2.0:1789760769.418429:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23646208->23650303] (req 23646208->23650303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.418442:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23646208->23650303] (req 23646208->23650303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.418452:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23646208->23650303] (req 23646208->23650303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.418459:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760769.418463:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23646208->23650303] (req 23646208->23650303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.418473:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23646208->23650303] (req 23646208->23650303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.418491:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23646208->23650303] (req 23646208->23650303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.418499:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209798 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23646208->23650303] (req 23646208->23650303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760769.418595:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abb100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484307584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50252us (50529us total) trans 4294973091 rc 0/0 00000100:00100000:2.0:1789760769.418605:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5802 00000100:00100000:2.0:1789760769.418615:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758648566412 time 758699259463 state 0x2 ref 0 00000100:00100000:0.0:1789760769.420683:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508934272:0@lo:41:osp-pre-1-0.0 00000100:00100000:0.0:1789760769.420749:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760769.420862:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508934272 00000100:00100000:1.0:1789760769.420870:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508934272 00000100:00100000:1.0:1789760769.420919:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 462 00000100:00100000:1.0:1789760769.420928:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508934272:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760769.420966:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760769.421026:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0000 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508934272:12345-0@lo:41:osp-pre-1-0.0 Request processed in 98us (277us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760769.421033:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 462 00000100:00100000:1.0:1789760769.421089:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508934272:0@lo:41:osp-pre-1-0.0 00000100:00100000:2.0:1789760769.426639:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760769.426698:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484307712 00000100:00100000:3.0:1789760769.426710:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484307712 00000100:00100000:3.0:1789760769.426763:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758648566412 time 758699259463 state 0x0 ref 1 00000100:00100000:3.0:1789760769.426779:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758699519879 time 758699259463 state 0x1 ref 1 00000100:00100000:3.0:1789760769.426809:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758699519879 time 758757739288 state 0x0 ref 1 00000100:00100000:3.0:1789760769.426822:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484307712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760769.426870:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23650304, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:3.0:1789760769.426879:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.426904:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:3.0:1789760769.426909:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760769.426916:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23650304->23654399] (req 23650304->23654399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.426930:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23650304->23654399] (req 23650304->23654399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760769.477423:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973092 is committed 00002000:00010000:2.0:1789760769.477471:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23650304 -> 23654400 00010000:00010000:2.0:1789760769.477486:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23650304->23654399] (req 23650304->23654399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.477499:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23650304->23654399] (req 23650304->23654399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.477509:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23650304->23654399] (req 23650304->23654399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.477516:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760769.477526:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23650304->23654399] (req 23650304->23654399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.477536:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23650304->23654399] (req 23650304->23654399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.477559:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23650304->23654399] (req 23650304->23654399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.477573:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620979f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23650304->23654399] (req 23650304->23654399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760769.478277:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484307712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50841us (51035us total) trans 4294973092 rc 0/0 00000100:00100000:2.0:1789760769.478289:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5803 00000100:00100000:2.0:1789760769.478299:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758699519879 time 758757739288 state 0x2 ref 0 00000100:00100000:2.0:1789760769.484881:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760769.484975:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484307840 00000100:00100000:0.0:1789760769.484985:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484307840 00000100:00100000:0.0:1789760769.485043:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758699519879 time 758757739288 state 0x0 ref 1 00000100:00100000:0.0:1789760769.485057:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758757999704 time 758757739288 state 0x1 ref 1 00000100:00100000:0.0:1789760769.485172:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758757999704 time 758816101658 state 0x0 ref 1 00000100:00100000:0.0:1789760769.485186:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484307840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760769.485234:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23654400, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:0.0:1789760769.485244:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.485261:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:0.0:1789760769.485265:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760769.485272:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23654400->23658495] (req 23654400->23658495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.485291:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23654400->23658495] (req 23654400->23658495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760769.517729:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23654400 -> 23658496 00010000:00010000:2.0:1789760769.517745:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23654400->23658495] (req 23654400->23658495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.517758:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23654400->23658495] (req 23654400->23658495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.517767:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23654400->23658495] (req 23654400->23658495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.517774:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760769.517779:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23654400->23658495] (req 23654400->23658495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.517787:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23654400->23658495] (req 23654400->23658495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.517809:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23654400->23658495] (req 23654400->23658495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.517816:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf38263962097a6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23654400->23658495] (req 23654400->23658495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760769.517914:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484307840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32727us (33037us total) trans 4294973093 rc 0/0 00000100:00100000:2.0:1789760769.517924:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5804 00000100:00100000:2.0:1789760769.517932:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758757999704 time 758816101658 state 0x2 ref 0 00000001:00080000:1.0:1789760769.519137:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973093 is committed 00000100:00100000:0.0:1789760769.527727:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760769.528135:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484307968 00000100:00100000:2.0:1789760769.528145:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484307968 00000100:00100000:2.0:1789760769.528198:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758757999704 time 758816101658 state 0x0 ref 1 00000100:00100000:2.0:1789760769.528213:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758816362074 time 758816101658 state 0x1 ref 1 00000100:00100000:2.0:1789760769.528287:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758816362074 time 758859216858 state 0x0 ref 1 00000100:00100000:2.0:1789760769.528301:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484307968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760769.529382:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23658496, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:2.0:1789760769.529396:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.529412:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:2.0:1789760769.529417:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760769.529423:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23658496->23662591] (req 23658496->23662591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.529439:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23658496->23662591] (req 23658496->23662591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760769.570745:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973094 is committed 00002000:00010000:2.0:1789760769.571450:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23658496 -> 23662592 00010000:00010000:2.0:1789760769.571465:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23658496->23662591] (req 23658496->23662591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.571489:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23658496->23662591] (req 23658496->23662591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.571601:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23658496->23662591] (req 23658496->23662591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.571611:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760769.571615:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23658496->23662591] (req 23658496->23662591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.571626:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23658496->23662591] (req 23658496->23662591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.571646:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23658496->23662591] (req 23658496->23662591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.571655:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf38263962097ad lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23658496->23662591] (req 23658496->23662591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760769.571748:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484307968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43446us (44025us total) trans 4294973094 rc 0/0 00000100:00100000:2.0:1789760769.571758:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5805 00000100:00100000:2.0:1789760769.571768:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758816362074 time 758859216858 state 0x2 ref 0 00000100:00100000:2.0:1789760769.577713:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760769.580183:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484308096 00000100:00100000:0.0:1789760769.580197:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484308096 00000100:00100000:0.0:1789760769.580270:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758816362074 time 758859216858 state 0x0 ref 1 00000100:00100000:0.0:1789760769.580285:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758859477274 time 758859216858 state 0x1 ref 1 00000100:00100000:0.0:1789760769.580370:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758859477274 time 758911300215 state 0x0 ref 1 00000100:00100000:0.0:1789760769.580386:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484308096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760769.580432:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23662592, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:0.0:1789760769.580440:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.580454:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:0.0:1789760769.580458:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760769.580464:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23662592->23666687] (req 23662592->23666687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.580490:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23662592->23666687] (req 23662592->23666687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760769.621538:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23662592 -> 23666688 00010000:00010000:0.0:1789760769.621559:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23662592->23666687] (req 23662592->23666687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.621573:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23662592->23666687] (req 23662592->23666687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.621582:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23662592->23666687] (req 23662592->23666687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.621589:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760769.621594:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23662592->23666687] (req 23662592->23666687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.621603:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23662592->23666687] (req 23662592->23666687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.621623:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23662592->23666687] (req 23662592->23666687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.621631:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097b4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23662592->23666687] (req 23662592->23666687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760769.621730:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484308096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41344us (44022us total) trans 4294973095 rc 0/0 00000100:00100000:0.0:1789760769.621746:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5806 00000100:00100000:0.0:1789760769.621755:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758859477274 time 758911300215 state 0x2 ref 0 00000001:00080000:1.0:1789760769.623158:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973095 is committed 00000100:00100000:0.0:1789760769.626872:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760769.627019:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484308224 00000100:00100000:3.0:1789760769.627030:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484308224 00000100:00100000:0.0:1789760769.627403:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758859477274 time 758911300215 state 0x0 ref 1 00000100:00100000:0.0:1789760769.627419:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758911560631 time 758911300215 state 0x1 ref 1 00000100:00100000:0.0:1789760769.627450:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758911560631 time 758958381377 state 0x0 ref 1 00000100:00100000:0.0:1789760769.627465:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484308224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760769.627513:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23666688, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:0.0:1789760769.627522:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.627597:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=758s v=(0 0) 00010000:00010000:0.0:1789760769.627602:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760769.627609:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23666688->23670783] (req 23666688->23670783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.627625:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23666688->23670783] (req 23666688->23670783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760769.679822:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23666688 -> 23670784 00010000:00010000:2.0:1789760769.679843:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23666688->23670783] (req 23666688->23670783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.679857:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23666688->23670783] (req 23666688->23670783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.679867:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23666688->23670783] (req 23666688->23670783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.679875:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760769.679880:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23666688->23670783] (req 23666688->23670783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.679899:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23666688->23670783] (req 23666688->23670783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.679923:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23666688->23670783] (req 23666688->23670783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.679937:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf38263962097bb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23666688->23670783] (req 23666688->23670783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760769.680043:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484308224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52577us (53174us total) trans 4294973096 rc 0/0 00000100:00100000:2.0:1789760769.680054:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5807 00000100:00100000:2.0:1789760769.680115:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758911560631 time 758958381377 state 0x2 ref 0 00000001:00080000:1.0:1789760769.682406:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973096 is committed 00000100:00100000:2.0:1789760769.683092:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760769.683177:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484308352 00000100:00100000:0.0:1789760769.683187:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484308352 00000100:00100000:0.0:1789760769.683253:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758911560631 time 758958381377 state 0x0 ref 1 00000100:00100000:0.0:1789760769.683313:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758958641793 time 758958381377 state 0x1 ref 1 00000100:00100000:0.0:1789760769.683344:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758958641793 time 759014274338 state 0x0 ref 1 00000100:00100000:0.0:1789760769.683359:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484308352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760769.683408:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23670784, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:0.0:1789760769.683417:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.683433:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:0.0:1789760769.683437:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760769.683444:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23670784->23674879] (req 23670784->23674879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.683457:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23670784->23674879] (req 23670784->23674879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760769.717793:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973097 is committed 00002000:00010000:1.0:1789760769.718060:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23670784 -> 23674880 00010000:00010000:1.0:1789760769.718093:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23670784->23674879] (req 23670784->23674879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.718116:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23670784->23674879] (req 23670784->23674879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.718125:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23670784->23674879] (req 23670784->23674879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.718133:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760769.718137:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23670784->23674879] (req 23670784->23674879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.718146:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23670784->23674879] (req 23670784->23674879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.718165:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23670784->23674879] (req 23670784->23674879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.718172:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097c2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23670784->23674879] (req 23670784->23674879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760769.718304:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484308352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34944us (35212us total) trans 4294973097 rc 0/0 00000100:00100000:1.0:1789760769.718318:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5808 00000100:00100000:1.0:1789760769.718328:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758958641793 time 759014274338 state 0x2 ref 0 00000100:00100000:3.0:1789760769.723686:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760769.724088:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484308480 00000100:00100000:2.0:1789760769.724099:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484308480 00000100:00100000:2.0:1789760769.724159:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 758958641793 time 759014274338 state 0x0 ref 1 00000100:00100000:2.0:1789760769.724182:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759014534754 time 759014274338 state 0x1 ref 1 00000100:00100000:2.0:1789760769.724203:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759014534754 time 759055135089 state 0x0 ref 1 00000100:00100000:2.0:1789760769.724214:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8aba680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484308480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760769.724260:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23674880, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:2.0:1789760769.724269:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.724283:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:2.0:1789760769.724287:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760769.724294:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23674880->23678975] (req 23674880->23678975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.724308:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23674880->23678975] (req 23674880->23678975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760769.762221:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973098 is committed 00002000:00010000:3.0:1789760769.762324:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23674880 -> 23678976 00010000:00010000:3.0:1789760769.762341:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23674880->23678975] (req 23674880->23678975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.762354:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23674880->23678975] (req 23674880->23678975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.762364:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23674880->23678975] (req 23674880->23678975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.762371:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760769.762376:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23674880->23678975] (req 23674880->23678975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.762385:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23674880->23678975] (req 23674880->23678975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.762406:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23674880->23678975] (req 23674880->23678975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.762414:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf38263962097c9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23674880->23678975] (req 23674880->23678975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760769.762605:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8aba680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484308480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38387us (38923us total) trans 4294973098 rc 0/0 00000100:00100000:3.0:1789760769.762615:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5809 00000100:00100000:3.0:1789760769.762625:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759014534754 time 759055135089 state 0x2 ref 0 00000100:00100000:0.0:1789760769.771434:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760769.771595:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484308608 00000100:00100000:2.0:1789760769.771604:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484308608 00000100:00100000:2.0:1789760769.771652:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759014534754 time 759055135089 state 0x0 ref 1 00000100:00100000:2.0:1789760769.771668:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759055395505 time 759055135089 state 0x1 ref 1 00000100:00100000:2.0:1789760769.771873:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759055395505 time 759102802679 state 0x0 ref 1 00000100:00100000:2.0:1789760769.771954:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484308608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760769.772004:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23678976, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:2.0:1789760769.772012:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.772026:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:2.0:1789760769.772031:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760769.772036:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23678976->23683071] (req 23678976->23683071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.772050:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23678976->23683071] (req 23678976->23683071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760769.802709:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973099 is committed 00002000:00010000:0.0:1789760769.802823:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23678976 -> 23683072 00010000:00010000:0.0:1789760769.802839:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23678976->23683071] (req 23678976->23683071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.802853:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23678976->23683071] (req 23678976->23683071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.802868:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23678976->23683071] (req 23678976->23683071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.802876:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760769.802881:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23678976->23683071] (req 23678976->23683071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.802902:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23678976->23683071] (req 23678976->23683071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.802921:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23678976->23683071] (req 23678976->23683071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.802929:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf38263962097d0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23678976->23683071] (req 23678976->23683071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760769.803028:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484308608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31139us (31600us total) trans 4294973099 rc 0/0 00000100:00100000:0.0:1789760769.803038:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5810 00000100:00100000:0.0:1789760769.803048:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759055395505 time 759102802679 state 0x2 ref 0 00000100:00100000:2.0:1789760769.807585:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760769.807703:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484308736 00000100:00100000:0.0:1789760769.807713:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484308736 00000100:00100000:0.0:1789760769.807778:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759055395505 time 759102802679 state 0x0 ref 1 00000100:00100000:0.0:1789760769.807796:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759103063095 time 759102802679 state 0x1 ref 1 00000100:00100000:0.0:1789760769.807831:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759103063095 time 759138761902 state 0x0 ref 1 00000100:00100000:0.0:1789760769.807845:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484308736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760769.807910:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23683072, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:0.0:1789760769.807920:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.807936:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:0.0:1789760769.807941:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760769.807950:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23683072->23687167] (req 23683072->23687167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.807976:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23683072->23687167] (req 23683072->23687167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760769.843546:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973100 is committed 00002000:00010000:0.0:1789760769.843640:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23683072 -> 23687168 00010000:00010000:0.0:1789760769.843656:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23683072->23687167] (req 23683072->23687167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.843669:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23683072->23687167] (req 23683072->23687167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.843679:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23683072->23687167] (req 23683072->23687167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.843687:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760769.843693:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23683072->23687167] (req 23683072->23687167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.843703:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23683072->23687167] (req 23683072->23687167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.843721:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23683072->23687167] (req 23683072->23687167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.843729:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097d7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23683072->23687167] (req 23683072->23687167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760769.843853:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484308736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36007us (36274us total) trans 4294973100 rc 0/0 00000100:00100000:0.0:1789760769.843862:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5811 00000100:00100000:0.0:1789760769.843872:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759103063095 time 759138761902 state 0x2 ref 0 00000100:00100000:1.0:1789760769.855704:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760769.855798:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484308864 00000100:00100000:0.0:1789760769.855808:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484308864 00000100:00100000:0.0:1789760769.855880:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759103063095 time 759138761902 state 0x0 ref 1 00000100:00100000:0.0:1789760769.855896:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759139022318 time 759138761902 state 0x1 ref 1 00000100:00100000:0.0:1789760769.855931:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759139022318 time 759186860765 state 0x0 ref 1 00000100:00100000:0.0:1789760769.855944:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484308864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760769.855995:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23687168, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:0.0:1789760769.856004:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.856020:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:0.0:1789760769.856037:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760769.856044:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23687168->23691263] (req 23687168->23691263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.856059:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23687168->23691263] (req 23687168->23691263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760769.898182:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23687168 -> 23691264 00010000:00010000:0.0:1789760769.898203:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23687168->23691263] (req 23687168->23691263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.898217:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23687168->23691263] (req 23687168->23691263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.898227:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23687168->23691263] (req 23687168->23691263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.898235:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760769.898239:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23687168->23691263] (req 23687168->23691263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.898248:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23687168->23691263] (req 23687168->23691263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.898274:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23687168->23691263] (req 23687168->23691263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760769.898282:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097de lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23687168->23691263] (req 23687168->23691263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760769.898388:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484308864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42442us (42689us total) trans 4294973101 rc 0/0 00000100:00100000:0.0:1789760769.898399:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5812 00000100:00100000:0.0:1789760769.898408:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759139022318 time 759186860765 state 0x2 ref 0 00000001:00080000:2.0:1789760769.898531:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973101 is committed 00000100:00100000:2.0:1789760769.903470:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760769.903559:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484308992 00000100:00100000:3.0:1789760769.903570:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484308992 00000100:00100000:3.0:1789760769.903690:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759139022318 time 759186860765 state 0x0 ref 1 00000100:00100000:3.0:1789760769.903707:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759187121181 time 759186860765 state 0x1 ref 1 00000100:00100000:3.0:1789760769.903771:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759187121181 time 759234700945 state 0x0 ref 1 00000100:00100000:3.0:1789760769.903784:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484308992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760769.903836:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23691264, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:3.0:1789760769.903845:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.903861:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:3.0:1789760769.903866:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760769.903872:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23691264->23695359] (req 23691264->23695359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760769.903886:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23691264->23695359] (req 23691264->23695359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.933635:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:1.0:1789760769.959054:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23691264 -> 23695360 00010000:00010000:1.0:1789760769.959093:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23691264->23695359] (req 23691264->23695359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.959119:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23691264->23695359] (req 23691264->23695359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.959129:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23691264->23695359] (req 23691264->23695359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.959136:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760769.959141:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23691264->23695359] (req 23691264->23695359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.959150:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23691264->23695359] (req 23691264->23695359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.959170:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23691264->23695359] (req 23691264->23695359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760769.959178:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf38263962097e5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23691264->23695359] (req 23691264->23695359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760769.959276:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484308992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55489us (55811us total) trans 4294973102 rc 0/0 00000100:00100000:1.0:1789760769.959286:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5813 00000100:00100000:1.0:1789760769.959295:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759187121181 time 759234700945 state 0x2 ref 0 00000001:00080000:2.0:1789760769.959997:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973102 is committed 00000100:00100000:1.0:1789760769.963255:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760769.963349:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484309120 00000100:00100000:2.0:1789760769.963361:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484309120 00000100:00100000:2.0:1789760769.963414:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759187121181 time 759234700945 state 0x0 ref 1 00000100:00100000:2.0:1789760769.963427:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759234961361 time 759234700945 state 0x1 ref 1 00000100:00100000:2.0:1789760769.963447:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759234961361 time 759294378909 state 0x0 ref 1 00000100:00100000:2.0:1789760769.963459:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484309120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760769.963517:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23695360, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:2.0:1789760769.963527:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.963543:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:2.0:1789760769.963547:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760769.963552:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23695360->23699455] (req 23695360->23699455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760769.963565:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23695360->23699455] (req 23695360->23699455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760770.010710:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23695360 -> 23699456 00010000:00010000:3.0:1789760770.010731:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23695360->23699455] (req 23695360->23699455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.010744:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23695360->23699455] (req 23695360->23699455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.010754:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23695360->23699455] (req 23695360->23699455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.010762:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760770.010768:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23695360->23699455] (req 23695360->23699455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.010777:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23695360->23699455] (req 23695360->23699455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.010798:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23695360->23699455] (req 23695360->23699455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.010807:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf38263962097ec lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23695360->23699455] (req 23695360->23699455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760770.010913:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484309120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47451us (47720us total) trans 4294973103 rc 0/0 00000100:00100000:3.0:1789760770.010924:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5814 00000100:00100000:3.0:1789760770.010934:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759234961361 time 759294378909 state 0x2 ref 0 00000001:00080000:1.0:1789760770.011442:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973103 is committed 00000100:00100000:3.0:1789760770.020981:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760770.021050:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484309248 00000100:00100000:0.0:1789760770.021060:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484309248 00000100:00100000:0.0:1789760770.021265:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759234961361 time 759294378909 state 0x0 ref 1 00000100:00100000:0.0:1789760770.021281:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759294639325 time 759294378909 state 0x1 ref 1 00000100:00100000:0.0:1789760770.021310:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759294639325 time 759352241583 state 0x0 ref 1 00000100:00100000:0.0:1789760770.021328:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760770.021374:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23699456, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:0.0:1789760770.021383:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.021398:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:0.0:1789760770.021403:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760770.021409:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23699456->23703551] (req 23699456->23703551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.021423:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23699456->23703551] (req 23699456->23703551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760770.021768:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760770.051510:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973104 is committed 00002000:00010000:1.0:1789760770.051670:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23699456 -> 23703552 00010000:00010000:1.0:1789760770.051687:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23699456->23703551] (req 23699456->23703551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.051701:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23699456->23703551] (req 23699456->23703551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.051709:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23699456->23703551] (req 23699456->23703551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.051717:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760770.051721:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23699456->23703551] (req 23699456->23703551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.051730:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23699456->23703551] (req 23699456->23703551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.051749:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23699456->23703551] (req 23699456->23703551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.051759:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962097f3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23699456->23703551] (req 23699456->23703551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760770.052003:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30678us (31025us total) trans 4294973104 rc 0/0 00000100:00100000:1.0:1789760770.052010:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5815 00000100:00100000:1.0:1789760770.052018:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759294639325 time 759352241583 state 0x2 ref 0 00000100:00100000:2.0:1789760770.059001:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760770.059085:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484309376 00000100:00100000:0.0:1789760770.059092:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484309376 00000100:00100000:0.0:1789760770.059161:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759294639325 time 759352241583 state 0x0 ref 1 00000100:00100000:0.0:1789760770.059177:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759352501999 time 759352241583 state 0x1 ref 1 00000100:00100000:0.0:1789760770.059210:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759352501999 time 759390140489 state 0x0 ref 1 00000100:00100000:0.0:1789760770.059223:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760770.059275:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23703552, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:0.0:1789760770.059283:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.059299:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:0.0:1789760770.059303:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760770.059310:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23703552->23707647] (req 23703552->23707647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.059325:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23703552->23707647] (req 23703552->23707647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760770.113787:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973105 is committed 00002000:00010000:1.0:1789760770.113919:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23703552 -> 23707648 00010000:00010000:1.0:1789760770.113936:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23703552->23707647] (req 23703552->23707647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.113951:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23703552->23707647] (req 23703552->23707647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.113961:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23703552->23707647] (req 23703552->23707647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.113968:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760770.113973:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23703552->23707647] (req 23703552->23707647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.113982:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23703552->23707647] (req 23703552->23707647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.114008:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23703552->23707647] (req 23703552->23707647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.114015:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962097fa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23703552->23707647] (req 23703552->23707647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760770.114357:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55132us (55358us total) trans 4294973105 rc 0/0 00000100:00100000:1.0:1789760770.114368:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5816 00000100:00100000:1.0:1789760770.114377:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759352501999 time 759390140489 state 0x2 ref 0 00000100:00100000:2.0:1789760770.126205:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760770.126272:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484309504 00000100:00100000:1.0:1789760770.126283:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484309504 00000100:00100000:1.0:1789760770.126334:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759352501999 time 759390140489 state 0x0 ref 1 00000100:00100000:1.0:1789760770.126349:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759390400905 time 759390140489 state 0x1 ref 1 00000100:00100000:1.0:1789760770.126382:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759390400905 time 759457312829 state 0x0 ref 1 00000100:00100000:1.0:1789760770.126395:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760770.126456:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23707648, mtime: 0x6aad9501, atime: 0x0, ctime: 0x6aad9501, blocks: 0xb807 00010000:00010000:1.0:1789760770.126465:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.126481:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:1.0:1789760770.126486:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760770.126492:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23707648->23711743] (req 23707648->23711743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.126505:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23707648->23711743] (req 23707648->23711743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760770.162899:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973106 is committed 00002000:00010000:1.0:1789760770.163128:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23707648 -> 23711744 00002000:00010000:1.0:1789760770.163132:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760769 -> 1789760770 00002000:00010000:1.0:1789760770.163134:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760769 -> 1789760770 00010000:00010000:1.0:1789760770.163147:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23707648->23711743] (req 23707648->23711743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.163158:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23707648->23711743] (req 23707648->23711743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.163168:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23707648->23711743] (req 23707648->23711743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.163175:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760770.163180:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23707648->23711743] (req 23707648->23711743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.163189:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23707648->23711743] (req 23707648->23711743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.163208:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23707648->23711743] (req 23707648->23711743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.163217:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209801 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23707648->23711743] (req 23707648->23711743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760770.163306:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36909us (37114us total) trans 4294973106 rc 0/0 00000100:00100000:1.0:1789760770.163316:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5817 00000100:00100000:1.0:1789760770.163325:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759390400905 time 759457312829 state 0x2 ref 0 00000100:00100000:2.0:1789760770.169970:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760770.170062:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484309632 00000100:00100000:3.0:1789760770.170109:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484309632 00000100:00100000:3.0:1789760770.170170:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759390400905 time 759457312829 state 0x0 ref 1 00000100:00100000:3.0:1789760770.170185:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759457573245 time 759457312829 state 0x1 ref 1 00000100:00100000:3.0:1789760770.170225:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759457573245 time 759501156345 state 0x0 ref 1 00000100:00100000:3.0:1789760770.170237:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760770.170600:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23711744, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:3.0:1789760770.170611:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.170627:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:3.0:1789760770.170632:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760770.170638:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23711744->23715839] (req 23711744->23715839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.170652:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23711744->23715839] (req 23711744->23715839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760770.207425:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973107 is committed 00002000:00010000:1.0:1789760770.208278:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23711744 -> 23715840 00010000:00010000:1.0:1789760770.208296:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23711744->23715839] (req 23711744->23715839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.208311:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23711744->23715839] (req 23711744->23715839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.208320:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23711744->23715839] (req 23711744->23715839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.208328:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760770.208333:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23711744->23715839] (req 23711744->23715839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.208342:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23711744->23715839] (req 23711744->23715839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.208366:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23711744->23715839] (req 23711744->23715839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.208375:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209808 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23711744->23715839] (req 23711744->23715839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760770.208523:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38282us (38556us total) trans 4294973107 rc 0/0 00000100:00100000:1.0:1789760770.208535:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5818 00000100:00100000:1.0:1789760770.208546:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759457573245 time 759501156345 state 0x2 ref 0 00000100:00100000:2.0:1789760770.231044:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760770.231167:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484309760 00000100:00100000:0.0:1789760770.231178:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484309760 00000100:00100000:3.0:1789760770.231295:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759457573245 time 759501156345 state 0x0 ref 1 00000100:00100000:3.0:1789760770.231309:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759501416761 time 759501156345 state 0x1 ref 1 00000100:00100000:3.0:1789760770.231337:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759501416761 time 759562268458 state 0x0 ref 1 00000100:00100000:3.0:1789760770.231349:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760770.231394:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23715840, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:3.0:1789760770.231402:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.231418:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:3.0:1789760770.231423:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760770.231429:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23715840->23719935] (req 23715840->23719935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.231443:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23715840->23719935] (req 23715840->23719935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760770.290682:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973108 is committed 00002000:00010000:3.0:1789760770.291176:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23715840 -> 23719936 00010000:00010000:3.0:1789760770.291192:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23715840->23719935] (req 23715840->23719935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.291207:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23715840->23719935] (req 23715840->23719935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.291216:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23715840->23719935] (req 23715840->23719935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.291223:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760770.291228:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23715840->23719935] (req 23715840->23719935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.291236:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23715840->23719935] (req 23715840->23719935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.291256:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23715840->23719935] (req 23715840->23719935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.291265:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620980f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23715840->23719935] (req 23715840->23719935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760770.291364:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484309760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60012us (60326us total) trans 4294973108 rc 0/0 00000100:00100000:3.0:1789760770.291374:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5819 00000100:00100000:3.0:1789760770.291384:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759501416761 time 759562268458 state 0x2 ref 0 00000100:00100000:2.0:1789760770.297027:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760770.297156:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484309888 00000100:00100000:1.0:1789760770.297167:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484309888 00000100:00100000:1.0:1789760770.297230:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759501416761 time 759562268458 state 0x0 ref 1 00000100:00100000:1.0:1789760770.297245:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759562528874 time 759562268458 state 0x1 ref 1 00000100:00100000:1.0:1789760770.297273:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759562528874 time 759628203859 state 0x0 ref 1 00000100:00100000:1.0:1789760770.297286:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484309888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760770.297336:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23719936, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:1.0:1789760770.297345:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.297360:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:1.0:1789760770.297367:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760770.297373:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23719936->23724031] (req 23719936->23724031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.297386:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23719936->23724031] (req 23719936->23724031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760770.327856:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973109 is committed 00002000:00010000:0.0:1789760770.327890:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23719936 -> 23724032 00010000:00010000:0.0:1789760770.327905:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23719936->23724031] (req 23719936->23724031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.327917:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23719936->23724031] (req 23719936->23724031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.327925:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23719936->23724031] (req 23719936->23724031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.327932:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760770.327937:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23719936->23724031] (req 23719936->23724031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.327947:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23719936->23724031] (req 23719936->23724031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.327966:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23719936->23724031] (req 23719936->23724031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.327975:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209816 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23719936->23724031] (req 23719936->23724031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760770.328219:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484309888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30930us (31197us total) trans 4294973109 rc 0/0 00000100:00100000:0.0:1789760770.328229:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5820 00000100:00100000:0.0:1789760770.328238:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759562528874 time 759628203859 state 0x2 ref 0 00000100:00100000:2.0:1789760770.336315:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760770.338474:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484310016 00000100:00100000:1.0:1789760770.338490:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484310016 00000100:00100000:1.0:1789760770.338561:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759562528874 time 759628203859 state 0x0 ref 1 00000100:00100000:1.0:1789760770.338576:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759628464275 time 759628203859 state 0x1 ref 1 00000100:00100000:1.0:1789760770.338612:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759628464275 time 759669542703 state 0x0 ref 1 00000100:00100000:1.0:1789760770.338625:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484310016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760770.338673:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23724032, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:1.0:1789760770.338682:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.338696:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:1.0:1789760770.338700:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760770.338707:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23724032->23728127] (req 23724032->23728127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.338719:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23724032->23728127] (req 23724032->23728127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760770.382203:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973110 is committed 00002000:00010000:1.0:1789760770.383246:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23724032 -> 23728128 00010000:00010000:1.0:1789760770.383265:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23724032->23728127] (req 23724032->23728127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.383279:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23724032->23728127] (req 23724032->23728127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.383287:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23724032->23728127] (req 23724032->23728127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.383295:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760770.383300:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23724032->23728127] (req 23724032->23728127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.383309:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23724032->23728127] (req 23724032->23728127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.383328:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23724032->23728127] (req 23724032->23728127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.383337:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620981d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23724032->23728127] (req 23724032->23728127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760770.383440:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484310016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44811us (47128us total) trans 4294973110 rc 0/0 00000100:00100000:1.0:1789760770.383455:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5821 00000100:00100000:1.0:1789760770.383465:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759628464275 time 759669542703 state 0x2 ref 0 00000100:00100000:2.0:1789760770.389201:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760770.389251:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484310144 00000100:00100000:3.0:1789760770.389261:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484310144 00000100:00100000:3.0:1789760770.389316:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759628464275 time 759669542703 state 0x0 ref 1 00000100:00100000:3.0:1789760770.389333:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759669803119 time 759669542703 state 0x1 ref 1 00000100:00100000:3.0:1789760770.389359:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759669803119 time 759720290517 state 0x0 ref 1 00000100:00100000:3.0:1789760770.389372:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484310144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760770.389422:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23728128, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:3.0:1789760770.389430:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.389446:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:3.0:1789760770.389450:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760770.389455:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23728128->23732223] (req 23728128->23732223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.389467:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23728128->23732223] (req 23728128->23732223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760770.446460:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973111 is committed 00002000:00010000:3.0:1789760770.446525:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23728128 -> 23732224 00010000:00010000:3.0:1789760770.446540:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23728128->23732223] (req 23728128->23732223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.446554:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23728128->23732223] (req 23728128->23732223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.446562:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23728128->23732223] (req 23728128->23732223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.446570:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760770.446574:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23728128->23732223] (req 23728128->23732223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.446583:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23728128->23732223] (req 23728128->23732223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.446604:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23728128->23732223] (req 23728128->23732223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.446611:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209824 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23728128->23732223] (req 23728128->23732223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760770.446709:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484310144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57325us (57512us total) trans 4294973111 rc 0/0 00000100:00100000:3.0:1789760770.446720:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5822 00000100:00100000:3.0:1789760770.446729:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759669803119 time 759720290517 state 0x2 ref 0 00000100:00100000:1.0:1789760770.450841:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760770.451008:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484310272 00000100:00100000:3.0:1789760770.451017:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484310272 00000100:00100000:3.0:1789760770.451094:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759669803119 time 759720290517 state 0x0 ref 1 00000100:00100000:3.0:1789760770.451120:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759720550933 time 759720290517 state 0x1 ref 1 00000100:00100000:3.0:1789760770.451145:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759720550933 time 759782077027 state 0x0 ref 1 00000100:00100000:3.0:1789760770.451158:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484310272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760770.451219:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23732224, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:3.0:1789760770.451228:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.451243:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:3.0:1789760770.451247:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760770.451253:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23732224->23736319] (req 23732224->23736319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.451268:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23732224->23736319] (req 23732224->23736319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760770.497579:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23732224 -> 23736320 00010000:00010000:0.0:1789760770.497599:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23732224->23736319] (req 23732224->23736319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.497612:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23732224->23736319] (req 23732224->23736319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.497621:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23732224->23736319] (req 23732224->23736319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.497629:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760770.497633:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23732224->23736319] (req 23732224->23736319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.497642:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23732224->23736319] (req 23732224->23736319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.497661:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23732224->23736319] (req 23732224->23736319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.497674:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620982b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23732224->23736319] (req 23732224->23736319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760770.497861:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973112 is committed 00000100:00100000:0.0:1789760770.497986:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484310272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46824us (47148us total) trans 4294973112 rc 0/0 00000100:00100000:0.0:1789760770.497997:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5823 00000100:00100000:0.0:1789760770.498007:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759720550933 time 759782077027 state 0x2 ref 0 00000100:00100000:3.0:1789760770.504137:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760770.504252:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484310400 00000100:00100000:0.0:1789760770.504263:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484310400 00000100:00100000:0.0:1789760770.504330:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759720550933 time 759782077027 state 0x0 ref 1 00000100:00100000:0.0:1789760770.504346:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759782337443 time 759782077027 state 0x1 ref 1 00000100:00100000:0.0:1789760770.504376:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759782337443 time 759835307222 state 0x0 ref 1 00000100:00100000:0.0:1789760770.504391:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484310400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760770.504440:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23736320, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:0.0:1789760770.504449:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.504465:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:0.0:1789760770.504470:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760770.504477:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23736320->23740415] (req 23736320->23740415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.504497:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23736320->23740415] (req 23736320->23740415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760770.553375:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973113 is committed 00002000:00010000:0.0:1789760770.553614:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23736320 -> 23740416 00010000:00010000:0.0:1789760770.553630:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23736320->23740415] (req 23736320->23740415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.553643:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23736320->23740415] (req 23736320->23740415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.553653:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23736320->23740415] (req 23736320->23740415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.553661:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760770.553667:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23736320->23740415] (req 23736320->23740415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.553676:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23736320->23740415] (req 23736320->23740415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.553696:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23736320->23740415] (req 23736320->23740415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.553704:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209832 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23736320->23740415] (req 23736320->23740415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760770.553806:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484310400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49414us (49674us total) trans 4294973113 rc 0/0 00000100:00100000:0.0:1789760770.553818:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5824 00000100:00100000:0.0:1789760770.553828:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759782337443 time 759835307222 state 0x2 ref 0 00000100:00100000:3.0:1789760770.559196:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760770.559384:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484310528 00000100:00100000:1.0:1789760770.559395:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484310528 00000100:00100000:1.0:1789760770.559456:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759782337443 time 759835307222 state 0x0 ref 1 00000100:00100000:1.0:1789760770.559473:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759835567638 time 759835307222 state 0x1 ref 1 00000100:00100000:1.0:1789760770.559497:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759835567638 time 759890428333 state 0x0 ref 1 00000100:00100000:1.0:1789760770.559510:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484310528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760770.559559:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23740416, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:1.0:1789760770.559568:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.559584:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:1.0:1789760770.559589:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760770.559595:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23740416->23744511] (req 23740416->23744511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.559609:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23740416->23744511] (req 23740416->23744511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760770.588818:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973114 is committed 00002000:00010000:0.0:1789760770.588964:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23740416 -> 23744512 00010000:00010000:0.0:1789760770.588981:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23740416->23744511] (req 23740416->23744511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.588994:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23740416->23744511] (req 23740416->23744511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.589002:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23740416->23744511] (req 23740416->23744511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.589010:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760770.589015:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23740416->23744511] (req 23740416->23744511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.589024:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23740416->23744511] (req 23740416->23744511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.589044:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23740416->23744511] (req 23740416->23744511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.589056:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209839 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23740416->23744511] (req 23740416->23744511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760770.589393:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484310528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29879us (30212us total) trans 4294973114 rc 0/0 00000100:00100000:0.0:1789760770.589403:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5825 00000100:00100000:0.0:1789760770.589412:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759835567638 time 759890428333 state 0x2 ref 0 00000100:00100000:1.0:1789760770.594536:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760770.594832:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484310656 00000100:00100000:3.0:1789760770.594841:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484310656 00000100:00100000:3.0:1789760770.594904:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759835567638 time 759890428333 state 0x0 ref 1 00000100:00100000:3.0:1789760770.594920:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759890688749 time 759890428333 state 0x1 ref 1 00000100:00100000:3.0:1789760770.594951:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759890688749 time 759925882228 state 0x0 ref 1 00000100:00100000:3.0:1789760770.594963:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484310656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760770.595011:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23744512, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:3.0:1789760770.595020:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.595035:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:3.0:1789760770.595040:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760770.595046:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23744512->23748607] (req 23744512->23748607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.595058:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23744512->23748607] (req 23744512->23748607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760770.636751:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973115 is committed 00002000:00010000:3.0:1789760770.637109:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23744512 -> 23748608 00010000:00010000:3.0:1789760770.637124:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23744512->23748607] (req 23744512->23748607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.637137:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23744512->23748607] (req 23744512->23748607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.637151:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23744512->23748607] (req 23744512->23748607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.637159:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760770.637164:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23744512->23748607] (req 23744512->23748607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.637173:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23744512->23748607] (req 23744512->23748607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.637192:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23744512->23748607] (req 23744512->23748607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.637199:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209840 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23744512->23748607] (req 23744512->23748607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760770.637443:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484310656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42476us (42911us total) trans 4294973115 rc 0/0 00000100:00100000:3.0:1789760770.637453:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5826 00000100:00100000:3.0:1789760770.637463:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759890688749 time 759925882228 state 0x2 ref 0 00000100:00100000:3.0:1789760770.646535:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760770.646675:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484310784 00000100:00100000:0.0:1789760770.646686:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484310784 00000100:00100000:0.0:1789760770.646747:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759890688749 time 759925882228 state 0x0 ref 1 00000100:00100000:0.0:1789760770.646771:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759926142644 time 759925882228 state 0x1 ref 1 00000100:00100000:0.0:1789760770.646804:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759926142644 time 759977735444 state 0x0 ref 1 00000100:00100000:0.0:1789760770.646818:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484310784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760770.646865:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23748608, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:0.0:1789760770.646874:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.646891:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=759s v=(0 0) 00010000:00010000:0.0:1789760770.646896:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760770.646902:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23748608->23752703] (req 23748608->23752703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.646921:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23748608->23752703] (req 23748608->23752703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760770.718379:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23748608 -> 23752704 00010000:00010000:0.0:1789760770.718400:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23748608->23752703] (req 23748608->23752703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.718413:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23748608->23752703] (req 23748608->23752703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.718421:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23748608->23752703] (req 23748608->23752703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.718428:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760770.718433:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23748608->23752703] (req 23748608->23752703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.718442:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23748608->23752703] (req 23748608->23752703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.718461:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23748608->23752703] (req 23748608->23752703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.718469:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209847 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23748608->23752703] (req 23748608->23752703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760770.718555:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484310784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 71736us (72025us total) trans 4294973116 rc 0/0 00000100:00100000:0.0:1789760770.718567:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5827 00000100:00100000:0.0:1789760770.718576:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759926142644 time 759977735444 state 0x2 ref 0 00000001:00080000:3.0:1789760770.719349:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973116 is committed 00000100:00100000:2.0:1789760770.730631:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760770.730743:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484310912 00000100:00100000:0.0:1789760770.730754:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484310912 00000100:00100000:0.0:1789760770.730814:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759926142644 time 759977735444 state 0x0 ref 1 00000100:00100000:0.0:1789760770.730831:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759977995860 time 759977735444 state 0x1 ref 1 00000100:00100000:0.0:1789760770.730865:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759977995860 time 760061796476 state 0x0 ref 1 00000100:00100000:0.0:1789760770.730880:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484310912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760770.730941:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23752704, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:0.0:1789760770.730951:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.730967:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760770.730972:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760770.730978:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23752704->23756799] (req 23752704->23756799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.730994:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23752704->23756799] (req 23752704->23756799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760770.769682:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973117 is committed 00002000:00010000:0.0:1789760770.769715:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23752704 -> 23756800 00010000:00010000:0.0:1789760770.769731:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23752704->23756799] (req 23752704->23756799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.769743:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23752704->23756799] (req 23752704->23756799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.769752:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23752704->23756799] (req 23752704->23756799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.769760:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760770.769765:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23752704->23756799] (req 23752704->23756799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.769774:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23752704->23756799] (req 23752704->23756799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.769793:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23752704->23756799] (req 23752704->23756799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.769800:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620984e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23752704->23756799] (req 23752704->23756799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760770.769899:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484310912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39017us (39272us total) trans 4294973117 rc 0/0 00000100:00100000:0.0:1789760770.769912:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5828 00000100:00100000:0.0:1789760770.769922:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759977995860 time 760061796476 state 0x2 ref 0 00000100:00100000:1.0:1789760770.774313:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760770.774623:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484311040 00000100:00100000:3.0:1789760770.774634:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484311040 00000100:00100000:3.0:1789760770.774690:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 759977995860 time 760061796476 state 0x0 ref 1 00000100:00100000:3.0:1789760770.774705:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760062056892 time 760061796476 state 0x1 ref 1 00000100:00100000:3.0:1789760770.774730:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760062056892 time 760105661855 state 0x0 ref 1 00000100:00100000:3.0:1789760770.774743:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484311040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760770.774793:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23756800, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:3.0:1789760770.774803:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.774819:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:3.0:1789760770.774823:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760770.774829:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23756800->23760895] (req 23756800->23760895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.774843:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23756800->23760895] (req 23756800->23760895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760770.814804:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973118 is committed 00002000:00010000:2.0:1789760770.814900:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23756800 -> 23760896 00010000:00010000:2.0:1789760770.814916:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23756800->23760895] (req 23756800->23760895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.814928:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23756800->23760895] (req 23756800->23760895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.814938:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23756800->23760895] (req 23756800->23760895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.814944:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760770.814948:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23756800->23760895] (req 23756800->23760895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.814962:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23756800->23760895] (req 23756800->23760895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.814982:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23756800->23760895] (req 23756800->23760895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.814995:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209855 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23756800->23760895] (req 23756800->23760895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760770.815230:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484311040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40484us (40921us total) trans 4294973118 rc 0/0 00000100:00100000:2.0:1789760770.815240:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5829 00000100:00100000:2.0:1789760770.815250:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760062056892 time 760105661855 state 0x2 ref 0 00000100:00100000:3.0:1789760770.824868:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760770.824987:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484311168 00000100:00100000:0.0:1789760770.824997:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484311168 00000100:00100000:0.0:1789760770.825061:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760062056892 time 760105661855 state 0x0 ref 1 00000100:00100000:0.0:1789760770.825136:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760105922271 time 760105661855 state 0x1 ref 1 00000100:00100000:0.0:1789760770.825168:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760105922271 time 760156099325 state 0x0 ref 1 00000100:00100000:0.0:1789760770.825183:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484311168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760770.825233:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23760896, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:0.0:1789760770.825243:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.825259:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760770.825263:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760770.825270:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23760896->23764991] (req 23760896->23764991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.825283:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23760896->23764991] (req 23760896->23764991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760770.868275:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973119 is committed 00002000:00010000:2.0:1789760770.868313:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23760896 -> 23764992 00010000:00010000:2.0:1789760770.868328:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23760896->23764991] (req 23760896->23764991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.868341:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23760896->23764991] (req 23760896->23764991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.868350:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23760896->23764991] (req 23760896->23764991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.868357:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760770.868362:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23760896->23764991] (req 23760896->23764991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.868371:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23760896->23764991] (req 23760896->23764991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.868390:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23760896->23764991] (req 23760896->23764991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.868397:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620985c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23760896->23764991] (req 23760896->23764991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760770.868486:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484311168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43302us (43623us total) trans 4294973119 rc 0/0 00000100:00100000:2.0:1789760770.868497:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5830 00000100:00100000:2.0:1789760770.868506:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760105922271 time 760156099325 state 0x2 ref 0 00000100:00100000:3.0:1789760770.872560:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760770.872643:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484311296 00000100:00100000:0.0:1789760770.872655:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484311296 00000100:00100000:0.0:1789760770.872735:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760105922271 time 760156099325 state 0x0 ref 1 00000100:00100000:0.0:1789760770.872753:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760156359741 time 760156099325 state 0x1 ref 1 00000100:00100000:0.0:1789760770.872783:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760156359741 time 760203713910 state 0x0 ref 1 00000100:00100000:0.0:1789760770.872797:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484311296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760770.872847:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23764992, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:0.0:1789760770.872856:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.872872:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760770.872877:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760770.872884:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23764992->23769087] (req 23764992->23769087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.872898:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23764992->23769087] (req 23764992->23769087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760770.909421:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973120 is committed 00002000:00010000:1.0:1789760770.909548:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23764992 -> 23769088 00010000:00010000:1.0:1789760770.909563:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23764992->23769087] (req 23764992->23769087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.909575:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23764992->23769087] (req 23764992->23769087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.909584:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23764992->23769087] (req 23764992->23769087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.909592:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760770.909597:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23764992->23769087] (req 23764992->23769087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.909606:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23764992->23769087] (req 23764992->23769087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.909624:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23764992->23769087] (req 23764992->23769087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760770.909631:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209863 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23764992->23769087] (req 23764992->23769087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760770.909725:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484311296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36927us (37170us total) trans 4294973120 rc 0/0 00000100:00100000:1.0:1789760770.909740:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5831 00000100:00100000:1.0:1789760770.909750:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760156359741 time 760203713910 state 0x2 ref 0 00000100:00100000:1.0:1789760770.918330:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760770.918436:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484311424 00000100:00100000:2.0:1789760770.918446:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484311424 00000100:00100000:2.0:1789760770.918496:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760156359741 time 760203713910 state 0x0 ref 1 00000100:00100000:2.0:1789760770.918510:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760203974326 time 760203713910 state 0x1 ref 1 00000100:00100000:2.0:1789760770.918544:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760203974326 time 760249475837 state 0x0 ref 1 00000100:00100000:2.0:1789760770.918564:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484311424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760770.918611:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23769088, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:2.0:1789760770.918620:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.918636:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:2.0:1789760770.918640:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760770.918646:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23769088->23773183] (req 23769088->23773183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760770.918659:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23769088->23773183] (req 23769088->23773183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760770.954246:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23769088 -> 23773184 00010000:00010000:3.0:1789760770.954265:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23769088->23773183] (req 23769088->23773183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.954279:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23769088->23773183] (req 23769088->23773183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.954422:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23769088->23773183] (req 23769088->23773183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.954430:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760770.954434:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23769088->23773183] (req 23769088->23773183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.954442:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23769088->23773183] (req 23769088->23773183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.954461:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23769088->23773183] (req 23769088->23773183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.954469:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620986a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23769088->23773183] (req 23769088->23773183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760770.954564:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484311424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36004us (36238us total) trans 4294973121 rc 0/0 00000100:00100000:3.0:1789760770.954575:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5832 00000100:00100000:3.0:1789760770.954584:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760203974326 time 760249475837 state 0x2 ref 0 00000001:00080000:1.0:1789760770.955379:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973121 is committed 00000100:00100000:3.0:1789760770.959319:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760770.959364:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484311552 00000100:00100000:3.0:1789760770.959373:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484311552 00000100:00100000:3.0:1789760770.959425:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760203974326 time 760249475837 state 0x0 ref 1 00000100:00100000:3.0:1789760770.959443:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760249736253 time 760249475837 state 0x1 ref 1 00000100:00100000:3.0:1789760770.959466:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760249736253 time 760290396940 state 0x0 ref 1 00000100:00100000:3.0:1789760770.959478:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abb800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484311552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760770.959526:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23773184, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:3.0:1789760770.959535:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.959550:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:3.0:1789760770.959555:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760770.959560:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23773184->23777279] (req 23773184->23777279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760770.959574:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23773184->23777279] (req 23773184->23777279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760770.960459:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:0.0:1789760771.005766:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973122 is committed 00002000:00010000:2.0:1789760771.005902:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23773184 -> 23777280 00010000:00010000:2.0:1789760771.005917:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23773184->23777279] (req 23773184->23777279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.005939:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23773184->23777279] (req 23773184->23777279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.005948:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23773184->23777279] (req 23773184->23777279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.005956:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760771.005961:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23773184->23777279] (req 23773184->23777279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.005971:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23773184->23777279] (req 23773184->23777279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.005996:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23773184->23777279] (req 23773184->23777279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.006003:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209871 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23773184->23777279] (req 23773184->23777279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760771.006145:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abb800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484311552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46663us (46828us total) trans 4294973122 rc 0/0 00000100:00100000:2.0:1789760771.006155:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5833 00000100:00100000:2.0:1789760771.006165:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760249736253 time 760290396940 state 0x2 ref 0 00000100:00100000:0.0:1789760771.015258:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760771.015359:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484311680 00000100:00100000:2.0:1789760771.015369:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484311680 00000100:00100000:2.0:1789760771.015421:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760249736253 time 760290396940 state 0x0 ref 1 00000100:00100000:2.0:1789760771.015437:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760290657356 time 760290396940 state 0x1 ref 1 00000100:00100000:2.0:1789760771.015469:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760290657356 time 760346399763 state 0x0 ref 1 00000100:00100000:2.0:1789760771.015481:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484311680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760771.015529:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23777280, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:2.0:1789760771.015538:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.015555:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:2.0:1789760771.015560:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760771.015566:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23777280->23781375] (req 23777280->23781375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.015580:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23777280->23781375] (req 23777280->23781375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760771.045720:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23777280 -> 23781376 00010000:00010000:0.0:1789760771.045741:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23777280->23781375] (req 23777280->23781375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.045760:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23777280->23781375] (req 23777280->23781375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.045768:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23777280->23781375] (req 23777280->23781375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.045776:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760771.045781:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23777280->23781375] (req 23777280->23781375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.045790:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23777280->23781375] (req 23777280->23781375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.045808:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23777280->23781375] (req 23777280->23781375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.045815:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209878 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23777280->23781375] (req 23777280->23781375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760771.045914:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484311680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30430us (30661us total) trans 4294973123 rc 0/0 00000100:00100000:0.0:1789760771.045931:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5834 00000100:00100000:0.0:1789760771.045941:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760290657356 time 760346399763 state 0x2 ref 0 00000001:00080000:3.0:1789760771.046109:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973123 is committed 00000100:00100000:1.0:1789760771.055972:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760771.056413:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484311808 00000100:00100000:2.0:1789760771.056423:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484311808 00000100:00100000:2.0:1789760771.056482:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760290657356 time 760346399763 state 0x0 ref 1 00000100:00100000:2.0:1789760771.056501:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760346660179 time 760346399763 state 0x1 ref 1 00000100:00100000:2.0:1789760771.056538:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760346660179 time 760387468857 state 0x0 ref 1 00000100:00100000:2.0:1789760771.056551:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9651c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484311808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760771.056601:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23781376, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:2.0:1789760771.056610:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.056624:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:2.0:1789760771.056628:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760771.056639:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23781376->23785471] (req 23781376->23785471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.056654:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23781376->23785471] (req 23781376->23785471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760771.099151:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973124 is committed 00002000:00010000:1.0:1789760771.100123:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23781376 -> 23785472 00010000:00010000:1.0:1789760771.100139:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23781376->23785471] (req 23781376->23785471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.100152:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23781376->23785471] (req 23781376->23785471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.100161:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23781376->23785471] (req 23781376->23785471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.100168:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760771.100173:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23781376->23785471] (req 23781376->23785471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.100183:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23781376->23785471] (req 23781376->23785471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.100203:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23781376->23785471] (req 23781376->23785471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.100216:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620987f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23781376->23785471] (req 23781376->23785471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760771.100353:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9651c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484311808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43799us (44384us total) trans 4294973124 rc 0/0 00000100:00100000:1.0:1789760771.100363:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5835 00000100:00100000:1.0:1789760771.100373:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760346660179 time 760387468857 state 0x2 ref 0 00000100:00100000:1.0:1789760771.105673:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760771.105762:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484311936 00000100:00100000:3.0:1789760771.105773:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484311936 00000100:00100000:3.0:1789760771.105827:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760346660179 time 760387468857 state 0x0 ref 1 00000100:00100000:3.0:1789760771.105843:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760387729273 time 760387468857 state 0x1 ref 1 00000100:00100000:3.0:1789760771.106021:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760387729273 time 760436806770 state 0x0 ref 1 00000100:00100000:3.0:1789760771.106040:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484311936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760771.107667:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23785472, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:3.0:1789760771.107682:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.107699:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:3.0:1789760771.107704:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760771.107710:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23785472->23789567] (req 23785472->23789567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.107725:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23785472->23789567] (req 23785472->23789567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760771.146402:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23785472 -> 23789568 00010000:00010000:2.0:1789760771.146948:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23785472->23789567] (req 23785472->23789567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.146964:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23785472->23789567] (req 23785472->23789567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.146974:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23785472->23789567] (req 23785472->23789567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.146982:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760771.146987:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23785472->23789567] (req 23785472->23789567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.146995:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23785472->23789567] (req 23785472->23789567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.147015:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23785472->23789567] (req 23785472->23789567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.147022:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209886 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23785472->23789567] (req 23785472->23789567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760771.148009:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484311936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41963us (42335us total) trans 4294973125 rc 0/0 00000100:00100000:2.0:1789760771.148024:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5836 00000100:00100000:2.0:1789760771.148034:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760387729273 time 760436806770 state 0x2 ref 0 00000001:00080000:2.0:1789760771.148712:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973125 is committed 00000100:00100000:0.0:1789760771.155609:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760771.157455:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484312064 00000100:00100000:3.0:1789760771.157470:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484312064 00000100:00100000:3.0:1789760771.157532:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760387729273 time 760436806770 state 0x0 ref 1 00000100:00100000:3.0:1789760771.157546:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760437067186 time 760436806770 state 0x1 ref 1 00000100:00100000:3.0:1789760771.157638:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760437067186 time 760488568295 state 0x0 ref 1 00000100:00100000:3.0:1789760771.157651:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484312064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760771.157720:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23789568, mtime: 0x6aad9502, atime: 0x0, ctime: 0x6aad9502, blocks: 0xb807 00010000:00010000:3.0:1789760771.157729:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.157751:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:3.0:1789760771.157756:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760771.157763:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23789568->23793663] (req 23789568->23793663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.157777:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23789568->23793663] (req 23789568->23793663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760771.190145:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23789568 -> 23793664 00002000:00010000:3.0:1789760771.190154:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760770 -> 1789760771 00002000:00010000:3.0:1789760771.190157:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760770 -> 1789760771 00010000:00010000:3.0:1789760771.190169:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23789568->23793663] (req 23789568->23793663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.190182:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23789568->23793663] (req 23789568->23793663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.190190:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23789568->23793663] (req 23789568->23793663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.190197:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760771.190201:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23789568->23793663] (req 23789568->23793663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.190210:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23789568->23793663] (req 23789568->23793663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.190229:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23789568->23793663] (req 23789568->23793663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.190236:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620988d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23789568->23793663] (req 23789568->23793663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760771.190332:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484312064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32678us (34726us total) trans 4294973126 rc 0/0 00000100:00100000:3.0:1789760771.190342:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5837 00000100:00100000:3.0:1789760771.190352:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760437067186 time 760488568295 state 0x2 ref 0 00000001:00080000:0.0:1789760771.191003:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973126 is committed 00000100:00100000:1.0:1789760771.195523:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760771.195670:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484312192 00000100:00100000:0.0:1789760771.195680:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484312192 00000100:00100000:0.0:1789760771.195736:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760437067186 time 760488568295 state 0x0 ref 1 00000100:00100000:0.0:1789760771.195752:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760488828711 time 760488568295 state 0x1 ref 1 00000100:00100000:0.0:1789760771.195785:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760488828711 time 760526715925 state 0x0 ref 1 00000100:00100000:0.0:1789760771.195798:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9652300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484312192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760771.195847:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23793664, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760771.195857:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.195873:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760771.195888:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760771.195895:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23793664->23797759] (req 23793664->23797759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.195910:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23793664->23797759] (req 23793664->23797759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760771.245544:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973127 is committed 00002000:00010000:0.0:1789760771.245622:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23793664 -> 23797760 00010000:00010000:0.0:1789760771.245638:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23793664->23797759] (req 23793664->23797759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.245650:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23793664->23797759] (req 23793664->23797759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.245659:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23793664->23797759] (req 23793664->23797759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.245667:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760771.245672:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23793664->23797759] (req 23793664->23797759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.245681:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23793664->23797759] (req 23793664->23797759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.245701:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23793664->23797759] (req 23793664->23797759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.245709:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209894 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23793664->23797759] (req 23793664->23797759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760771.245850:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9652300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484312192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50050us (50332us total) trans 4294973127 rc 0/0 00000100:00100000:0.0:1789760771.245860:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5838 00000100:00100000:0.0:1789760771.245868:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760488828711 time 760526715925 state 0x2 ref 0 00000100:00100000:0.0:1789760771.249633:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760771.249723:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484312320 00000100:00100000:1.0:1789760771.249733:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484312320 00000100:00100000:1.0:1789760771.249784:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760488828711 time 760526715925 state 0x0 ref 1 00000100:00100000:1.0:1789760771.249798:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760526976341 time 760526715925 state 0x1 ref 1 00000100:00100000:1.0:1789760771.249826:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760526976341 time 760580757832 state 0x0 ref 1 00000100:00100000:1.0:1789760771.249837:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706ec50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484312320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760771.249894:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23797760, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:1.0:1789760771.249903:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.249918:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:1.0:1789760771.249923:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760771.249929:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23797760->23801855] (req 23797760->23801855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.249943:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23797760->23801855] (req 23797760->23801855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760771.292968:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973128 is committed 00002000:00010000:1.0:1789760771.293141:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23797760 -> 23801856 00010000:00010000:1.0:1789760771.293158:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23797760->23801855] (req 23797760->23801855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.293171:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23797760->23801855] (req 23797760->23801855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.293189:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23797760->23801855] (req 23797760->23801855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.293197:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760771.293202:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23797760->23801855] (req 23797760->23801855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.293211:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23797760->23801855] (req 23797760->23801855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.293232:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23797760->23801855] (req 23797760->23801855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.293240:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620989b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23797760->23801855] (req 23797760->23801855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760771.293334:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706ec50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484312320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43492us (43706us total) trans 4294973128 rc 0/0 00000100:00100000:1.0:1789760771.293345:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5839 00000100:00100000:1.0:1789760771.293355:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760526976341 time 760580757832 state 0x2 ref 0 00000100:00100000:2.0:1789760771.297515:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760771.297617:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484312448 00000100:00100000:0.0:1789760771.297628:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484312448 00000100:00100000:0.0:1789760771.297705:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760526976341 time 760580757832 state 0x0 ref 1 00000100:00100000:0.0:1789760771.297729:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760581018248 time 760580757832 state 0x1 ref 1 00000100:00100000:0.0:1789760771.297798:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760581018248 time 760628728165 state 0x0 ref 1 00000100:00100000:0.0:1789760771.297812:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484312448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760771.297943:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23801856, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760771.297954:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.297973:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760771.297979:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760771.297986:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23801856->23805951] (req 23801856->23805951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.298003:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23801856->23805951] (req 23801856->23805951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760771.334625:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973129 is committed 00002000:00010000:1.0:1789760771.335680:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23801856 -> 23805952 00010000:00010000:1.0:1789760771.335698:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23801856->23805951] (req 23801856->23805951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.335711:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23801856->23805951] (req 23801856->23805951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.335720:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23801856->23805951] (req 23801856->23805951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.335727:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760771.335732:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23801856->23805951] (req 23801856->23805951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.335741:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23801856->23805951] (req 23801856->23805951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.335758:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23801856->23805951] (req 23801856->23805951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.335771:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf38263962098a2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23801856->23805951] (req 23801856->23805951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760771.336025:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484312448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38211us (38514us total) trans 4294973129 rc 0/0 00000100:00100000:1.0:1789760771.336035:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5840 00000100:00100000:1.0:1789760771.336044:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760581018248 time 760628728165 state 0x2 ref 0 00000100:00100000:2.0:1789760771.340323:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760771.340409:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484312576 00000100:00100000:3.0:1789760771.340421:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484312576 00000100:00100000:3.0:1789760771.340472:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760581018248 time 760628728165 state 0x0 ref 1 00000100:00100000:3.0:1789760771.340487:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760628988581 time 760628728165 state 0x1 ref 1 00000100:00100000:3.0:1789760771.340552:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760628988581 time 760671482411 state 0x0 ref 1 00000100:00100000:3.0:1789760771.340565:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484312576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760771.340617:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23805952, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:3.0:1789760771.340626:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.340646:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:3.0:1789760771.340650:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760771.340657:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23805952->23810047] (req 23805952->23810047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.340670:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23805952->23810047] (req 23805952->23810047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760771.375056:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973130 is committed 00002000:00010000:3.0:1789760771.375304:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23805952 -> 23810048 00010000:00010000:3.0:1789760771.375318:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23805952->23810047] (req 23805952->23810047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.375330:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23805952->23810047] (req 23805952->23810047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.375339:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23805952->23810047] (req 23805952->23810047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.375345:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760771.375350:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23805952->23810047] (req 23805952->23810047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.375358:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23805952->23810047] (req 23805952->23810047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.375376:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23805952->23810047] (req 23805952->23810047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.375383:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf38263962098a9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23805952->23810047] (req 23805952->23810047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760771.375476:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484312576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34909us (35412us total) trans 4294973130 rc 0/0 00000100:00100000:3.0:1789760771.375486:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5841 00000100:00100000:3.0:1789760771.375495:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760628988581 time 760671482411 state 0x2 ref 0 00000100:00100000:2.0:1789760771.379539:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760771.379630:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484312704 00000100:00100000:0.0:1789760771.379640:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484312704 00000100:00100000:0.0:1789760771.379700:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760628988581 time 760671482411 state 0x0 ref 1 00000100:00100000:0.0:1789760771.379715:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760671742827 time 760671482411 state 0x1 ref 1 00000100:00100000:0.0:1789760771.379744:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760671742827 time 760710675625 state 0x0 ref 1 00000100:00100000:0.0:1789760771.379755:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484312704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760771.379801:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23810048, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760771.379808:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.379823:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760771.379827:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760771.379833:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23810048->23814143] (req 23810048->23814143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.379845:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23810048->23814143] (req 23810048->23814143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760771.412378:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973131 is committed 00002000:00010000:0.0:1789760771.412515:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23810048 -> 23814144 00010000:00010000:0.0:1789760771.412531:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23810048->23814143] (req 23810048->23814143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.412544:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23810048->23814143] (req 23810048->23814143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.412553:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23810048->23814143] (req 23810048->23814143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.412561:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760771.412566:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23810048->23814143] (req 23810048->23814143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.412575:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23810048->23814143] (req 23810048->23814143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.412596:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23810048->23814143] (req 23810048->23814143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.412605:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098b0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23810048->23814143] (req 23810048->23814143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760771.412704:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484312704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32945us (33170us total) trans 4294973131 rc 0/0 00000100:00100000:0.0:1789760771.412714:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5842 00000100:00100000:0.0:1789760771.412724:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760671742827 time 760710675625 state 0x2 ref 0 00000100:00100000:2.0:1789760771.416473:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760771.416680:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484312832 00000100:00100000:1.0:1789760771.416689:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484312832 00000100:00100000:1.0:1789760771.416738:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760671742827 time 760710675625 state 0x0 ref 1 00000100:00100000:1.0:1789760771.416750:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760710936041 time 760710675625 state 0x1 ref 1 00000100:00100000:1.0:1789760771.416778:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760710936041 time 760747709572 state 0x0 ref 1 00000100:00100000:1.0:1789760771.416789:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484312832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760771.416832:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23814144, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:1.0:1789760771.416841:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.416855:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:1.0:1789760771.416859:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760771.416866:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23814144->23818239] (req 23814144->23818239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.416889:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23814144->23818239] (req 23814144->23818239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760771.449747:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23814144 -> 23818240 00010000:00010000:3.0:1789760771.449766:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23814144->23818239] (req 23814144->23818239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.449779:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23814144->23818239] (req 23814144->23818239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.449788:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23814144->23818239] (req 23814144->23818239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.449796:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760771.449801:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23814144->23818239] (req 23814144->23818239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.449810:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23814144->23818239] (req 23814144->23818239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.449833:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23814144->23818239] (req 23814144->23818239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.449841:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf38263962098b7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23814144->23818239] (req 23814144->23818239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760771.449931:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484312832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33139us (33463us total) trans 4294973132 rc 0/0 00000100:00100000:3.0:1789760771.449940:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5843 00000100:00100000:3.0:1789760771.449950:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760710936041 time 760747709572 state 0x2 ref 0 00000001:00080000:0.0:1789760771.450174:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973132 is committed 00000100:00100000:2.0:1789760771.453398:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760771.453678:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484312960 00000100:00100000:0.0:1789760771.453688:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484312960 00000100:00100000:0.0:1789760771.453749:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760710936041 time 760747709572 state 0x0 ref 1 00000100:00100000:0.0:1789760771.453763:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760747969988 time 760747709572 state 0x1 ref 1 00000100:00100000:0.0:1789760771.453796:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760747969988 time 760784727332 state 0x0 ref 1 00000100:00100000:0.0:1789760771.453810:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484312960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760771.453858:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23818240, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760771.453867:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.453898:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760771.453902:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760771.453908:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23818240->23822335] (req 23818240->23822335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.453922:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23818240->23822335] (req 23818240->23822335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760771.485658:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973133 is committed 00002000:00010000:0.0:1789760771.485719:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23818240 -> 23822336 00010000:00010000:0.0:1789760771.485733:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23818240->23822335] (req 23818240->23822335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.485747:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23818240->23822335] (req 23818240->23822335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.485756:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23818240->23822335] (req 23818240->23822335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.485764:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760771.485768:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23818240->23822335] (req 23818240->23822335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.485777:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23818240->23822335] (req 23818240->23822335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.485802:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23818240->23822335] (req 23818240->23822335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.485810:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098be lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23818240->23822335] (req 23818240->23822335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760771.485913:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484312960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32101us (32520us total) trans 4294973133 rc 0/0 00000100:00100000:0.0:1789760771.485924:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5844 00000100:00100000:0.0:1789760771.485933:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760747969988 time 760784727332 state 0x2 ref 0 00000100:00100000:3.0:1789760771.488914:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760771.488999:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484313088 00000100:00100000:0.0:1789760771.489009:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484313088 00000100:00100000:0.0:1789760771.489084:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760747969988 time 760784727332 state 0x0 ref 1 00000100:00100000:0.0:1789760771.489111:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760784987748 time 760784727332 state 0x1 ref 1 00000100:00100000:0.0:1789760771.489136:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760784987748 time 760820067529 state 0x0 ref 1 00000100:00100000:0.0:1789760771.489151:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484313088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760771.489210:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23822336, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760771.489218:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.489232:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760771.489237:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760771.489243:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23822336->23826431] (req 23822336->23826431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.489257:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23822336->23826431] (req 23822336->23826431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760771.526614:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973134 is committed 00002000:00010000:1.0:1789760771.526733:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23822336 -> 23826432 00010000:00010000:1.0:1789760771.526750:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23822336->23826431] (req 23822336->23826431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.526762:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23822336->23826431] (req 23822336->23826431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.526770:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23822336->23826431] (req 23822336->23826431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.526777:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760771.526781:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23822336->23826431] (req 23822336->23826431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.526790:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23822336->23826431] (req 23822336->23826431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.526811:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23822336->23826431] (req 23822336->23826431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.526819:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098c5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23822336->23826431] (req 23822336->23826431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760771.527837:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484313088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38684us (38925us total) trans 4294973134 rc 0/0 00000100:00100000:1.0:1789760771.527850:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5845 00000100:00100000:1.0:1789760771.527860:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760784987748 time 760820067529 state 0x2 ref 0 00000100:00100000:2.0:1789760771.532682:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760771.533604:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484313216 00000100:00100000:0.0:1789760771.533615:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484313216 00000100:00100000:0.0:1789760771.533676:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760784987748 time 760820067529 state 0x0 ref 1 00000100:00100000:0.0:1789760771.533691:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760820327945 time 760820067529 state 0x1 ref 1 00000100:00100000:0.0:1789760771.533757:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760820327945 time 760864687891 state 0x0 ref 1 00000100:00100000:0.0:1789760771.533772:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484313216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760771.533817:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23826432, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760771.533825:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.533841:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760771.533845:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760771.533851:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23826432->23830527] (req 23826432->23830527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.533864:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23826432->23830527] (req 23826432->23830527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760771.566860:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973135 is committed 00002000:00010000:1.0:1789760771.567083:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23826432 -> 23830528 00010000:00010000:1.0:1789760771.567107:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23826432->23830527] (req 23826432->23830527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.567120:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23826432->23830527] (req 23826432->23830527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.567130:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23826432->23830527] (req 23826432->23830527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.567138:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760771.567143:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23826432->23830527] (req 23826432->23830527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.567151:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23826432->23830527] (req 23826432->23830527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.567169:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23826432->23830527] (req 23826432->23830527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.567257:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf38263962098cc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23826432->23830527] (req 23826432->23830527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760771.567455:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484313216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33682us (34777us total) trans 4294973135 rc 0/0 00000100:00100000:1.0:1789760771.567465:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5846 00000100:00100000:1.0:1789760771.567475:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760820327945 time 760864687891 state 0x2 ref 0 00000100:00100000:3.0:1789760771.571615:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760771.571928:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484313344 00000100:00100000:0.0:1789760771.571937:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484313344 00000100:00100000:0.0:1789760771.571994:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760820327945 time 760864687891 state 0x0 ref 1 00000100:00100000:0.0:1789760771.572008:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760864948307 time 760864687891 state 0x1 ref 1 00000100:00100000:0.0:1789760771.572232:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760864948307 time 760903160806 state 0x0 ref 1 00000100:00100000:0.0:1789760771.572248:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484313344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760771.572297:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23830528, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760771.572305:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.572322:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:0.0:1789760771.572326:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760771.572333:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23830528->23834623] (req 23830528->23834623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.572346:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23830528->23834623] (req 23830528->23834623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760771.612393:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23830528 -> 23834624 00010000:00010000:1.0:1789760771.612413:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23830528->23834623] (req 23830528->23834623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.612426:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23830528->23834623] (req 23830528->23834623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.612434:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23830528->23834623] (req 23830528->23834623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.612442:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760771.612446:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23830528->23834623] (req 23830528->23834623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.612455:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23830528->23834623] (req 23830528->23834623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.612480:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23830528->23834623] (req 23830528->23834623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.612487:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf38263962098d3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23830528->23834623] (req 23830528->23834623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760771.612622:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484313344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40374us (41009us total) trans 4294973136 rc 0/0 00000100:00100000:1.0:1789760771.612633:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5847 00000100:00100000:1.0:1789760771.612643:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760864948307 time 760903160806 state 0x2 ref 0 00000001:00080000:0.0:1789760771.613435:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973136 is committed 00000100:00100000:3.0:1789760771.616277:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760771.616526:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484313472 00000100:00100000:2.0:1789760771.616536:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484313472 00000100:00100000:2.0:1789760771.616593:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760864948307 time 760903160806 state 0x0 ref 1 00000100:00100000:2.0:1789760771.616608:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760903421222 time 760903160806 state 0x1 ref 1 00000100:00100000:2.0:1789760771.616639:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760903421222 time 760947569866 state 0x0 ref 1 00000100:00100000:2.0:1789760771.616650:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484313472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760771.616696:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23834624, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:2.0:1789760771.616705:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.616720:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:2.0:1789760771.616724:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760771.616731:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23834624->23838719] (req 23834624->23838719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.616744:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23834624->23838719] (req 23834624->23838719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760771.645504:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973137 is committed 00002000:00010000:3.0:1789760771.645677:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23834624 -> 23838720 00010000:00010000:3.0:1789760771.645692:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23834624->23838719] (req 23834624->23838719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.645705:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23834624->23838719] (req 23834624->23838719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.645715:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23834624->23838719] (req 23834624->23838719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.645722:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760771.645728:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23834624->23838719] (req 23834624->23838719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.645737:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23834624->23838719] (req 23834624->23838719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.645756:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23834624->23838719] (req 23834624->23838719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.645765:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf38263962098da lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23834624->23838719] (req 23834624->23838719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760771.645891:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484313472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29237us (29618us total) trans 4294973137 rc 0/0 00000100:00100000:3.0:1789760771.645901:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5848 00000100:00100000:3.0:1789760771.645911:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760903421222 time 760947569866 state 0x2 ref 0 00000100:00100000:3.0:1789760771.650189:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760771.650422:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484313600 00000100:00100000:1.0:1789760771.650432:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484313600 00000100:00100000:1.0:1789760771.650486:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760903421222 time 760947569866 state 0x0 ref 1 00000100:00100000:1.0:1789760771.650501:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760947830282 time 760947569866 state 0x1 ref 1 00000100:00100000:1.0:1789760771.650565:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760947830282 time 760981495967 state 0x0 ref 1 00000100:00100000:1.0:1789760771.650576:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484313600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760771.650617:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23838720, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:1.0:1789760771.650624:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.650637:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=760s v=(0 0) 00010000:00010000:1.0:1789760771.650640:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760771.650645:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23838720->23842815] (req 23838720->23842815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.650655:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23838720->23842815] (req 23838720->23842815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760771.693239:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973138 is committed 00002000:00010000:2.0:1789760771.693394:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23838720 -> 23842816 00010000:00010000:2.0:1789760771.693409:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23838720->23842815] (req 23838720->23842815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.693422:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23838720->23842815] (req 23838720->23842815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.693431:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23838720->23842815] (req 23838720->23842815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.693438:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760771.693442:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23838720->23842815] (req 23838720->23842815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.693452:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23838720->23842815] (req 23838720->23842815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.693474:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23838720->23842815] (req 23838720->23842815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.693481:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf38263962098e1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23838720->23842815] (req 23838720->23842815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760771.693570:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484313600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42991us (43387us total) trans 4294973138 rc 0/0 00000100:00100000:2.0:1789760771.693580:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5849 00000100:00100000:2.0:1789760771.693590:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760947830282 time 760981495967 state 0x2 ref 0 00000100:00100000:3.0:1789760771.696669:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760771.696762:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484313728 00000100:00100000:1.0:1789760771.696771:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484313728 00000100:00100000:1.0:1789760771.696821:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760947830282 time 760981495967 state 0x0 ref 1 00000100:00100000:1.0:1789760771.696835:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760981756383 time 760981495967 state 0x1 ref 1 00000100:00100000:1.0:1789760771.696864:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760981756383 time 761027795248 state 0x0 ref 1 00000100:00100000:1.0:1789760771.696876:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abb100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484313728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760771.696983:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23842816, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:1.0:1789760771.696992:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.697007:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:1.0:1789760771.697012:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760771.697019:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23842816->23846911] (req 23842816->23846911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.697034:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23842816->23846911] (req 23842816->23846911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760771.729525:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973139 is committed 00002000:00010000:2.0:1789760771.730672:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23842816 -> 23846912 00010000:00010000:2.0:1789760771.730690:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23842816->23846911] (req 23842816->23846911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.730703:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23842816->23846911] (req 23842816->23846911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.730712:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23842816->23846911] (req 23842816->23846911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.730719:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760771.730724:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23842816->23846911] (req 23842816->23846911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.730733:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23842816->23846911] (req 23842816->23846911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.730753:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23842816->23846911] (req 23842816->23846911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.730766:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf38263962098e8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23842816->23846911] (req 23842816->23846911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760771.731767:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abb100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484313728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34884us (35099us total) trans 4294973139 rc 0/0 00000100:00100000:2.0:1789760771.731780:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5850 00000100:00100000:2.0:1789760771.731791:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760981756383 time 761027795248 state 0x2 ref 0 00000100:00100000:3.0:1789760771.736643:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760771.736747:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484313856 00000100:00100000:0.0:1789760771.736757:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484313856 00000100:00100000:0.0:1789760771.736818:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 760981756383 time 761027795248 state 0x0 ref 1 00000100:00100000:0.0:1789760771.736834:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761028055664 time 761027795248 state 0x1 ref 1 00000100:00100000:0.0:1789760771.736864:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761028055664 time 761067795538 state 0x0 ref 1 00000100:00100000:0.0:1789760771.736883:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8aba680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484313856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760771.736932:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23846912, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760771.736943:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.736964:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:0.0:1789760771.736969:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760771.736975:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23846912->23851007] (req 23846912->23851007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.736990:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23846912->23851007] (req 23846912->23851007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760771.790380:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973140 is committed 00002000:00010000:0.0:1789760771.790510:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23846912 -> 23851008 00010000:00010000:0.0:1789760771.790524:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23846912->23851007] (req 23846912->23851007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.790537:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23846912->23851007] (req 23846912->23851007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.790551:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23846912->23851007] (req 23846912->23851007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.790558:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760771.790562:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23846912->23851007] (req 23846912->23851007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.790571:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23846912->23851007] (req 23846912->23851007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.790590:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23846912->23851007] (req 23846912->23851007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.790597:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf38263962098ef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23846912->23851007] (req 23846912->23851007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760771.790828:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8aba680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484313856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53943us (54187us total) trans 4294973140 rc 0/0 00000100:00100000:0.0:1789760771.790838:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5851 00000100:00100000:0.0:1789760771.790849:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761028055664 time 761067795538 state 0x2 ref 0 00000100:00100000:0.0:1789760771.793406:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760771.793608:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484313984 00000100:00100000:3.0:1789760771.793617:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484313984 00000100:00100000:3.0:1789760771.793685:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761028055664 time 761067795538 state 0x0 ref 1 00000100:00100000:3.0:1789760771.793701:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761068055954 time 761067795538 state 0x1 ref 1 00000100:00100000:3.0:1789760771.793730:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761068055954 time 761124661264 state 0x0 ref 1 00000100:00100000:3.0:1789760771.793742:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484313984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760771.793789:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23851008, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:3.0:1789760771.793798:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.793814:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:3.0:1789760771.793819:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760771.793824:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23851008->23855103] (req 23851008->23855103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.793837:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23851008->23855103] (req 23851008->23855103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760771.828852:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973141 is committed 00002000:00010000:1.0:1789760771.828962:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23851008 -> 23855104 00010000:00010000:1.0:1789760771.828977:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23851008->23855103] (req 23851008->23855103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.828991:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23851008->23855103] (req 23851008->23855103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.829000:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23851008->23855103] (req 23851008->23855103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.829007:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760771.829012:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23851008->23855103] (req 23851008->23855103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.829021:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23851008->23855103] (req 23851008->23855103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.829040:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23851008->23855103] (req 23851008->23855103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.829049:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf38263962098f6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23851008->23855103] (req 23851008->23855103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760771.829169:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484313984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35424us (35767us total) trans 4294973141 rc 0/0 00000100:00100000:1.0:1789760771.829180:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5852 00000100:00100000:1.0:1789760771.829190:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761068055954 time 761124661264 state 0x2 ref 0 00000100:00100000:3.0:1789760771.834017:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760771.834533:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484314112 00000100:00100000:2.0:1789760771.834543:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484314112 00000100:00100000:2.0:1789760771.834601:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761068055954 time 761124661264 state 0x0 ref 1 00000100:00100000:2.0:1789760771.834618:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761124921680 time 761124661264 state 0x1 ref 1 00000100:00100000:2.0:1789760771.834750:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761124921680 time 761165680254 state 0x0 ref 1 00000100:00100000:2.0:1789760771.834764:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484314112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760771.834813:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23855104, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:2.0:1789760771.834822:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.834837:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:2.0:1789760771.834842:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760771.834848:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23855104->23859199] (req 23855104->23859199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760771.834862:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23855104->23859199] (req 23855104->23859199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760771.868824:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23855104 -> 23859200 00010000:00010000:1.0:1789760771.868843:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23855104->23859199] (req 23855104->23859199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.868857:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23855104->23859199] (req 23855104->23859199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.868866:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23855104->23859199] (req 23855104->23859199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.868883:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760771.868888:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23855104->23859199] (req 23855104->23859199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.868897:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23855104->23859199] (req 23855104->23859199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.868916:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23855104->23859199] (req 23855104->23859199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.868923:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf38263962098fd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23855104->23859199] (req 23855104->23859199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760771.869025:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484314112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34259us (35013us total) trans 4294973142 rc 0/0 00000100:00100000:1.0:1789760771.869036:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5853 00000100:00100000:1.0:1789760771.869045:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761124921680 time 761165680254 state 0x2 ref 0 00000001:00080000:3.0:1789760771.870113:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973142 is committed 00000100:00100000:2.0:1789760771.877041:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760771.877159:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484314240 00000100:00100000:1.0:1789760771.877169:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484314240 00000100:00100000:1.0:1789760771.877226:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761124921680 time 761165680254 state 0x0 ref 1 00000100:00100000:1.0:1789760771.877241:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761165940670 time 761165680254 state 0x1 ref 1 00000100:00100000:1.0:1789760771.877268:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761165940670 time 761208199314 state 0x0 ref 1 00000100:00100000:1.0:1789760771.877279:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484314240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760771.877323:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23859200, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:1.0:1789760771.877332:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.877347:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:1.0:1789760771.877351:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760771.877357:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23859200->23863295] (req 23859200->23863295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.877370:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23859200->23863295] (req 23859200->23863295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760771.911396:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973143 is committed 00002000:00010000:1.0:1789760771.911541:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23859200 -> 23863296 00010000:00010000:1.0:1789760771.911557:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23859200->23863295] (req 23859200->23863295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.911569:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23859200->23863295] (req 23859200->23863295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.911578:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23859200->23863295] (req 23859200->23863295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.911584:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760771.911589:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23859200->23863295] (req 23859200->23863295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.911598:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23859200->23863295] (req 23859200->23863295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.911617:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23859200->23863295] (req 23859200->23863295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760771.911623:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209904 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23859200->23863295] (req 23859200->23863295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760771.911747:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484314240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34463us (34708us total) trans 4294973143 rc 0/0 00000100:00100000:1.0:1789760771.911770:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5854 00000100:00100000:1.0:1789760771.911779:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761165940670 time 761208199314 state 0x2 ref 0 00000100:00100000:3.0:1789760771.916333:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760771.916599:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484314368 00000100:00100000:0.0:1789760771.916609:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484314368 00000100:00100000:0.0:1789760771.916666:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761165940670 time 761208199314 state 0x0 ref 1 00000100:00100000:0.0:1789760771.916684:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761208459730 time 761208199314 state 0x1 ref 1 00000100:00100000:0.0:1789760771.916746:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761208459730 time 761247676650 state 0x0 ref 1 00000100:00100000:0.0:1789760771.916770:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484314368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760771.916816:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23863296, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760771.916824:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.916839:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:0.0:1789760771.916843:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760771.916848:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23863296->23867391] (req 23863296->23867391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.916860:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23863296->23867391] (req 23863296->23867391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760771.957228:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23863296 -> 23867392 00010000:00010000:0.0:1789760771.957249:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23863296->23867391] (req 23863296->23867391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.957263:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23863296->23867391] (req 23863296->23867391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.957272:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23863296->23867391] (req 23863296->23867391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.957284:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760771.957289:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23863296->23867391] (req 23863296->23867391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.957298:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23863296->23867391] (req 23863296->23867391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.957316:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23863296->23867391] (req 23863296->23867391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.957324:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620990b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23863296->23867391] (req 23863296->23867391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760771.957433:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484314368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40662us (41101us total) trans 4294973144 rc 0/0 00000100:00100000:0.0:1789760771.957452:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5855 00000100:00100000:0.0:1789760771.957462:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761208459730 time 761247676650 state 0x2 ref 0 00000001:00080000:3.0:1789760771.961213:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973144 is committed 00000100:00100000:2.0:1789760771.961381:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760771.961525:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484314496 00000100:00100000:3.0:1789760771.961535:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484314496 00000100:00100000:3.0:1789760771.961583:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761208459730 time 761247676650 state 0x0 ref 1 00000100:00100000:3.0:1789760771.961597:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761247937066 time 761247676650 state 0x1 ref 1 00000100:00100000:3.0:1789760771.961626:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761247937066 time 761292557307 state 0x0 ref 1 00000100:00100000:3.0:1789760771.961642:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484314496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760771.961684:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23867392, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:3.0:1789760771.961692:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.961708:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:3.0:1789760771.961713:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760771.961719:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23867392->23871487] (req 23867392->23871487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760771.961732:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23867392->23871487] (req 23867392->23871487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760771.980389:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760772.003133:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973145 is committed 00002000:00010000:2.0:1789760772.003192:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23867392 -> 23871488 00010000:00010000:2.0:1789760772.003207:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23867392->23871487] (req 23867392->23871487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.003220:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23867392->23871487] (req 23867392->23871487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.003228:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23867392->23871487] (req 23867392->23871487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.003234:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760772.003239:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23867392->23871487] (req 23867392->23871487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.003247:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23867392->23871487] (req 23867392->23871487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.003266:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23867392->23871487] (req 23867392->23871487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.003278:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209912 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23867392->23871487] (req 23867392->23871487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760772.003363:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484314496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41718us (41982us total) trans 4294973145 rc 0/0 00000100:00100000:2.0:1789760772.003373:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5856 00000100:00100000:2.0:1789760772.003382:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761247937066 time 761292557307 state 0x2 ref 0 00000100:00100000:2.0:1789760772.016464:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760772.016644:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484314624 00000100:00100000:1.0:1789760772.016654:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484314624 00000100:00100000:1.0:1789760772.016730:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761247937066 time 761292557307 state 0x0 ref 1 00000100:00100000:1.0:1789760772.016745:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761292817723 time 761292557307 state 0x1 ref 1 00000100:00100000:1.0:1789760772.016776:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761292817723 time 761347707145 state 0x0 ref 1 00000100:00100000:1.0:1789760772.016789:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484314624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760772.016841:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23871488, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:1.0:1789760772.016849:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.016865:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:1.0:1789760772.016869:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760772.016875:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23871488->23875583] (req 23871488->23875583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.016889:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23871488->23875583] (req 23871488->23875583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1789760772.017097:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760772.058345:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973146 is committed 00002000:00010000:3.0:1789760772.058563:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23871488 -> 23875584 00010000:00010000:3.0:1789760772.058580:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23871488->23875583] (req 23871488->23875583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.058652:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23871488->23875583] (req 23871488->23875583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.058662:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23871488->23875583] (req 23871488->23875583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.058670:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.058676:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23871488->23875583] (req 23871488->23875583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.058686:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23871488->23875583] (req 23871488->23875583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.058707:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23871488->23875583] (req 23871488->23875583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.058715:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209919 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23871488->23875583] (req 23871488->23875583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.058806:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484314624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42015us (42347us total) trans 4294973146 rc 0/0 00000100:00100000:3.0:1789760772.058817:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5857 00000100:00100000:3.0:1789760772.058827:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761292817723 time 761347707145 state 0x2 ref 0 00000100:00100000:3.0:1789760772.065888:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760772.066629:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484314752 00000100:00100000:1.0:1789760772.066640:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484314752 00000100:00100000:1.0:1789760772.066708:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761292817723 time 761347707145 state 0x0 ref 1 00000100:00100000:1.0:1789760772.066725:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761347967561 time 761347707145 state 0x1 ref 1 00000100:00100000:1.0:1789760772.066757:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761347967561 time 761397687413 state 0x0 ref 1 00000100:00100000:1.0:1789760772.066770:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484314752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760772.066821:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23875584, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:1.0:1789760772.066830:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.066846:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:1.0:1789760772.066850:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760772.066857:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23875584->23879679] (req 23875584->23879679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.066871:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23875584->23879679] (req 23875584->23879679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760772.105406:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973147 is committed 00002000:00010000:3.0:1789760772.105868:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23875584 -> 23879680 00010000:00010000:3.0:1789760772.105884:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23875584->23879679] (req 23875584->23879679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.105898:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23875584->23879679] (req 23875584->23879679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.105907:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23875584->23879679] (req 23875584->23879679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.105920:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.105925:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23875584->23879679] (req 23875584->23879679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.105935:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23875584->23879679] (req 23875584->23879679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.105955:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23875584->23879679] (req 23875584->23879679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.105963:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209920 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23875584->23879679] (req 23875584->23879679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.106038:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484314752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39267us (40155us total) trans 4294973147 rc 0/0 00000100:00100000:3.0:1789760772.106049:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5858 00000100:00100000:3.0:1789760772.106059:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761347967561 time 761397687413 state 0x2 ref 0 00000100:00100000:2.0:1789760772.109316:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760772.109445:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484314880 00000100:00100000:0.0:1789760772.109455:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484314880 00000100:00100000:0.0:1789760772.109511:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761347967561 time 761397687413 state 0x0 ref 1 00000100:00100000:0.0:1789760772.109526:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761397947829 time 761397687413 state 0x1 ref 1 00000100:00100000:0.0:1789760772.109586:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761397947829 time 761440516928 state 0x0 ref 1 00000100:00100000:0.0:1789760772.109601:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484314880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760772.109647:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23879680, mtime: 0x6aad9503, atime: 0x0, ctime: 0x6aad9503, blocks: 0xb807 00010000:00010000:0.0:1789760772.109657:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.109673:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:0.0:1789760772.109677:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760772.109684:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23879680->23883775] (req 23879680->23883775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.109698:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23879680->23883775] (req 23879680->23883775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760772.135592:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973148 is committed 00002000:00010000:0.0:1789760772.135682:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23879680 -> 23883776 00002000:00010000:0.0:1789760772.135687:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760771 -> 1789760772 00002000:00010000:0.0:1789760772.135690:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760771 -> 1789760772 00010000:00010000:0.0:1789760772.135705:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23879680->23883775] (req 23879680->23883775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.135719:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23879680->23883775] (req 23879680->23883775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.135728:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23879680->23883775] (req 23879680->23883775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.135737:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760772.135742:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23879680->23883775] (req 23879680->23883775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.135751:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23879680->23883775] (req 23879680->23883775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.135770:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23879680->23883775] (req 23879680->23883775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.135777:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209927 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23879680->23883775] (req 23879680->23883775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760772.136042:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484314880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26438us (26728us total) trans 4294973148 rc 0/0 00000100:00100000:0.0:1789760772.136053:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5859 00000100:00100000:0.0:1789760772.136107:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761397947829 time 761440516928 state 0x2 ref 0 00000100:00100000:2.0:1789760772.141308:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760772.141441:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484315008 00000100:00100000:3.0:1789760772.142907:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484315008 00000100:00100000:3.0:1789760772.142979:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761397947829 time 761440516928 state 0x0 ref 1 00000100:00100000:3.0:1789760772.142995:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761440777344 time 761440516928 state 0x1 ref 1 00000100:00100000:3.0:1789760772.143143:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761440777344 time 761474072634 state 0x0 ref 1 00000100:00100000:3.0:1789760772.143157:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484315008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760772.143207:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23883776, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:3.0:1789760772.143217:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.143243:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:3.0:1789760772.143248:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760772.143254:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23883776->23887871] (req 23883776->23887871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.143268:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23883776->23887871] (req 23883776->23887871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760772.192164:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973149 is committed 00002000:00010000:3.0:1789760772.192279:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23883776 -> 23887872 00010000:00010000:3.0:1789760772.192295:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23883776->23887871] (req 23883776->23887871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.192308:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23883776->23887871] (req 23883776->23887871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.192317:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23883776->23887871] (req 23883776->23887871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.192324:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.192328:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23883776->23887871] (req 23883776->23887871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.192338:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23883776->23887871] (req 23883776->23887871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.192356:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23883776->23887871] (req 23883776->23887871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.192365:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620992e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23883776->23887871] (req 23883776->23887871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.192458:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484315008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49299us (51153us total) trans 4294973149 rc 0/0 00000100:00100000:3.0:1789760772.192468:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5860 00000100:00100000:3.0:1789760772.192476:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761440777344 time 761474072634 state 0x2 ref 0 00000100:00100000:3.0:1789760772.195241:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760772.195415:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484315136 00000100:00100000:0.0:1789760772.195425:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484315136 00000100:00100000:0.0:1789760772.195483:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761440777344 time 761474072634 state 0x0 ref 1 00000100:00100000:0.0:1789760772.195499:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761474333050 time 761474072634 state 0x1 ref 1 00000100:00100000:0.0:1789760772.195680:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761474333050 time 761526608779 state 0x0 ref 1 00000100:00100000:0.0:1789760772.195696:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484315136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760772.195740:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23887872, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:0.0:1789760772.195749:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.195764:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:0.0:1789760772.195768:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760772.195775:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23887872->23891967] (req 23887872->23891967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.195788:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23887872->23891967] (req 23887872->23891967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760772.225190:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973150 is committed 00002000:00010000:0.0:1789760772.225305:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23887872 -> 23891968 00010000:00010000:0.0:1789760772.225322:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23887872->23891967] (req 23887872->23891967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.225336:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23887872->23891967] (req 23887872->23891967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.225345:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23887872->23891967] (req 23887872->23891967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.225353:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760772.225359:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23887872->23891967] (req 23887872->23891967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.225367:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23887872->23891967] (req 23887872->23891967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.225391:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23887872->23891967] (req 23887872->23891967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.225399:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209935 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23887872->23891967] (req 23887872->23891967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760772.225499:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484315136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29799us (30259us total) trans 4294973150 rc 0/0 00000100:00100000:0.0:1789760772.225509:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5861 00000100:00100000:0.0:1789760772.225518:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761474333050 time 761526608779 state 0x2 ref 0 00000100:00100000:2.0:1789760772.232267:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760772.232405:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484315264 00000100:00100000:1.0:1789760772.232415:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484315264 00000100:00100000:1.0:1789760772.232465:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761474333050 time 761526608779 state 0x0 ref 1 00000100:00100000:1.0:1789760772.232481:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761526869195 time 761526608779 state 0x1 ref 1 00000100:00100000:1.0:1789760772.232508:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761526869195 time 761563439746 state 0x0 ref 1 00000100:00100000:1.0:1789760772.232523:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484315264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760772.232571:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23891968, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:1.0:1789760772.232580:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.232596:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:1.0:1789760772.232600:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760772.232606:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23891968->23896063] (req 23891968->23896063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.232619:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23891968->23896063] (req 23891968->23896063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760772.319449:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23891968 -> 23896064 00010000:00010000:3.0:1789760772.319468:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23891968->23896063] (req 23891968->23896063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.319483:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23891968->23896063] (req 23891968->23896063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.319492:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23891968->23896063] (req 23891968->23896063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.319505:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.319510:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23891968->23896063] (req 23891968->23896063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.319518:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23891968->23896063] (req 23891968->23896063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.319537:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23891968->23896063] (req 23891968->23896063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.319546:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620993c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23891968->23896063] (req 23891968->23896063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.319640:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484315264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 87114us (87378us total) trans 4294973151 rc 0/0 00000100:00100000:3.0:1789760772.319652:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5862 00000100:00100000:3.0:1789760772.319661:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761526869195 time 761563439746 state 0x2 ref 0 00000001:00080000:2.0:1789760772.321131:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973151 is committed 00000100:00100000:2.0:1789760772.327223:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760772.327394:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484315392 00000100:00100000:1.0:1789760772.327404:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484315392 00000100:00100000:1.0:1789760772.327458:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761526869195 time 761563439746 state 0x0 ref 1 00000100:00100000:1.0:1789760772.327474:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761563700162 time 761563439746 state 0x1 ref 1 00000100:00100000:1.0:1789760772.327505:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761563700162 time 761658436311 state 0x0 ref 1 00000100:00100000:1.0:1789760772.327518:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484315392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760772.327565:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23896064, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:1.0:1789760772.327573:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.327608:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:1.0:1789760772.327612:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760772.327618:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23896064->23900159] (req 23896064->23900159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.327631:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23896064->23900159] (req 23896064->23900159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760772.398340:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23896064 -> 23900160 00010000:00010000:1.0:1789760772.398359:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23896064->23900159] (req 23896064->23900159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.398372:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23896064->23900159] (req 23896064->23900159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.398394:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23896064->23900159] (req 23896064->23900159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.398402:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760772.398407:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23896064->23900159] (req 23896064->23900159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.398417:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23896064->23900159] (req 23896064->23900159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.398438:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23896064->23900159] (req 23896064->23900159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.398447:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209943 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23896064->23900159] (req 23896064->23900159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760772.398545:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484315392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 71025us (71329us total) trans 4294973152 rc 0/0 00000100:00100000:1.0:1789760772.398557:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5863 00000100:00100000:1.0:1789760772.398568:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761563700162 time 761658436311 state 0x2 ref 0 00000001:00080000:2.0:1789760772.399897:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973152 is committed 00000100:00100000:0.0:1789760772.401783:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760772.401889:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484315520 00000100:00100000:3.0:1789760772.401900:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484315520 00000100:00100000:3.0:1789760772.401953:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761563700162 time 761658436311 state 0x0 ref 1 00000100:00100000:3.0:1789760772.401969:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761658696727 time 761658436311 state 0x1 ref 1 00000100:00100000:3.0:1789760772.402000:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761658696727 time 761732931882 state 0x0 ref 1 00000100:00100000:3.0:1789760772.402017:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484315520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760772.402732:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23900160, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:2.0:1789760772.402746:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.402763:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:2.0:1789760772.402769:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760772.402776:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23900160->23904255] (req 23900160->23904255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.402797:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23900160->23904255] (req 23900160->23904255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760772.444595:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973153 is committed 00002000:00010000:3.0:1789760772.444722:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23900160 -> 23904256 00010000:00010000:3.0:1789760772.444738:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23900160->23904255] (req 23900160->23904255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.444751:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23900160->23904255] (req 23900160->23904255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.444761:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23900160->23904255] (req 23900160->23904255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.444769:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.444774:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23900160->23904255] (req 23900160->23904255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.444783:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23900160->23904255] (req 23900160->23904255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.444802:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23900160->23904255] (req 23900160->23904255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.444810:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf382639620994a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23900160->23904255] (req 23900160->23904255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.444914:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484315520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42895us (43135us total) trans 4294973153 rc 0/0 00000100:00100000:3.0:1789760772.444925:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5864 00000100:00100000:3.0:1789760772.444935:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761658696727 time 761732931882 state 0x2 ref 0 00000100:00100000:0.0:1789760772.450426:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760772.450517:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484315648 00000100:00100000:3.0:1789760772.450527:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484315648 00000100:00100000:3.0:1789760772.450580:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761658696727 time 761732931882 state 0x0 ref 1 00000100:00100000:3.0:1789760772.450595:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761733192298 time 761732931882 state 0x1 ref 1 00000100:00100000:3.0:1789760772.450626:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761733192298 time 761781557183 state 0x0 ref 1 00000100:00100000:3.0:1789760772.450639:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484315648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760772.450696:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23904256, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:3.0:1789760772.450706:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.450722:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:3.0:1789760772.450726:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760772.450732:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23904256->23908351] (req 23904256->23908351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.450745:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23904256->23908351] (req 23904256->23908351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760772.492660:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9652d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508934400:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760772.492726:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760772.492785:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760772.492792:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9651c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508934528:0@lo:13:osp-pre-1-1.0 00000100:00100000:1.0:1789760772.492817:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760772.492857:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508934400 00000100:00100000:1.0:1789760772.492864:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508934400 00000100:00100000:1.0:1789760772.492921:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508934528 00000100:00100000:1.0:1789760772.492924:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508934528 00000100:00100000:1.0:1789760772.492941:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 404 00000100:00100000:1.0:1789760772.492949:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508934400:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:1.0:1789760772.492983:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760772.493026:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508934400:12345-0@lo:13:osp-pre-1-0.0 Request processed in 77us (300us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760772.493033:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 404 00000100:00100000:1.0:1789760772.493048:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 405 00000100:00100000:1.0:1789760772.493054:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9652300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508934528:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:1.0:1789760772.493101:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760772.493128:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9652300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508934528:12345-0@lo:13:osp-pre-1-1.0 Request processed in 74us (310us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760772.493134:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 405 00000100:00080000:1.0:1789760772.493153:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:3.0:1789760772.494562:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8ab9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3480:1876699508934656:0@lo:400:kworker.0 00000100:00100000:1.0:1789760772.494650:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9651c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508934528:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760772.495140:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8ab9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3481:1876699508934784:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.498200:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760772.498252:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f46d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508935168:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.498286:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760772.498306:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f47480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508935680:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.498328:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760772.498368:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508935168 00000100:00100000:3.0:1789760772.498375:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508935168 00000100:00100000:3.0:1789760772.498417:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 463 00000100:00100000:3.0:1789760772.498425:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508935168:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.498476:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508935168:12345-0@lo:400:kworker.0 Request processed in 52us (189us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760772.498484:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 463 00000100:00100000:3.0:1789760772.498506:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508935680 00000100:00100000:3.0:1789760772.498510:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508935680 00000100:00100000:3.0:1789760772.498555:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 341 00000100:00100000:3.0:1789760772.498563:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508935680:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.498596:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508935680:12345-0@lo:400:kworker.0 Request processed in 35us (267us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760772.498602:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 341 00000100:00100000:3.0:1789760772.498638:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f46d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508935168:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.498649:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f47480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508935680:0@lo:400:kworker.0 00000100:00100000:0.0:1789760772.500464:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508934656 00000100:00100000:0.0:1789760772.500475:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508934656 00000100:00100000:0.0:1789760772.500516:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 490 00000100:00100000:0.0:1789760772.500526:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8039450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3480:x1876699508934656:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.500621:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f45f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508935040:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.500681:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760772.500709:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f47b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508935552:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.500732:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760772.500754:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1789760772.500761:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8abb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508934912:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.500783:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760772.500800:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f45c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0000_UUID:3479:1876699508935424:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.500832:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760772.500851:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0001_UUID:3479:1876699508935936:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.500872:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760772.500911:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508935040 00000100:00100000:3.0:1789760772.500915:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508935040 00000100:00100000:3.0:1789760772.500942:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508935424 00000100:00100000:3.0:1789760772.500945:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508935424 00000100:00100000:3.0:1789760772.500965:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508935936 00000100:00100000:3.0:1789760772.500967:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508935936 00000100:00100000:3.0:1789760772.500991:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 975 00000100:00100000:3.0:1789760772.500998:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b82300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508935040:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.501053:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b82300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508935040:12345-0@lo:400:kworker.0 Request processed in 55us (372us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760772.501060:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 975 00000100:00100000:3.0:1789760772.501113:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 976 00000100:00100000:3.0:1789760772.501119:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b81180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0000_UUID+5:3479:x1876699508935424:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.501150:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b81180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0000_UUID+5:3479:x1876699508935424:12345-0@lo:400:kworker.0 Request processed in 32us (318us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760772.501156:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 976 00000100:00100000:3.0:1789760772.501167:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 977 00000100:00100000:3.0:1789760772.501172:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0001_UUID+5:3479:x1876699508935936:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.501198:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0001_UUID+5:3479:x1876699508935936:12345-0@lo:400:kworker.0 Request processed in 26us (325us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760772.501204:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 977 00000100:00100000:3.0:1789760772.501248:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f45c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0000_UUID:3479:1876699508935424:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.501257:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b80a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-lwp-OST0001_UUID:3479:1876699508935936:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.501265:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f45f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508935040:0@lo:400:kworker.0 00000100:00100000:0.0:1789760772.501563:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8039450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3480:x1876699508934656:12345-0@lo:400:kworker.0 Request processed in 1039us (3370us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760772.501569:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 490 00000100:00100000:0.0:1789760772.501600:0:11908:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508935552 00000100:00100000:0.0:1789760772.501604:0:11908:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508935552 00000100:00100000:0.0:1789760772.501636:0:11908:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 342 00000100:00100000:0.0:1789760772.501645:0:11908:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508935552:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760772.501682:0:11908:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_004:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508935552:12345-0@lo:400:kworker.0 Request processed in 39us (948us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760772.501689:0:11908:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 342 00000100:00100000:0.0:1789760772.501707:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508934912 00000100:00100000:0.0:1789760772.501711:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508934912 00000100:00100000:0.0:1789760772.501737:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 464 00000100:00100000:0.0:1789760772.501744:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508934912:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760772.501775:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508934912:12345-0@lo:400:kworker.0 Request processed in 33us (991us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760772.501780:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 464 00000100:00100000:0.0:1789760772.501824:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8abb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508934912:0@lo:400:kworker.0 00000100:00100000:0.0:1789760772.501835:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f47b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508935552:0@lo:400:kworker.0 00000100:00100000:3.0:1789760772.502421:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8ab9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3480:1876699508934656:0@lo:400:kworker.0 00000100:00100000:1.0:1789760772.502514:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9652d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508934400:0@lo:13:osp-pre-1-0.0 00000100:00080000:1.0:1789760772.502607:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:1.0:1789760772.502612:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760772.503267:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760772.503311:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f46680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3481:1876699508935296:0@lo:400:kworker.0 00000100:00100000:2.0:1789760772.503339:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760772.503367:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b83100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3481:1876699508935808:0@lo:400:kworker.0 00000100:00100000:2.0:1789760772.503390:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760772.503414:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508934784 00000100:00100000:2.0:1789760772.503420:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508934784 00000100:00100000:2.0:1789760772.503511:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508935296 00000100:00100000:2.0:1789760772.503514:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508935296 00000100:00100000:2.0:1789760772.503548:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508935808 00000100:00100000:2.0:1789760772.503552:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508935808 00000100:00100000:2.0:1789760772.503574:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 978 00000100:00100000:2.0:1789760772.503581:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3481:x1876699508934784:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760772.503625:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3481:x1876699508934784:12345-0@lo:400:kworker.0 Request processed in 44us (358us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760772.503632:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 978 00000100:00100000:2.0:1789760772.503649:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 979 00000100:00100000:2.0:1789760772.503655:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3481:x1876699508935296:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760772.503683:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3481:x1876699508935296:12345-0@lo:400:kworker.0 Request processed in 29us (343us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760772.503694:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 979 00000100:00100000:2.0:1789760772.503705:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 980 00000100:00100000:2.0:1789760772.503711:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3481:x1876699508935808:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760772.503738:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3481:x1876699508935808:12345-0@lo:400:kworker.0 Request processed in 28us (347us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760772.503743:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 980 00000100:00100000:2.0:1789760772.503775:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b83100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3481:1876699508935808:0@lo:400:kworker.0 00000100:00100000:2.0:1789760772.503795:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8ab9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3481:1876699508934784:0@lo:400:kworker.0 00000100:00100000:2.0:1789760772.503805:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f46680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3481:1876699508935296:0@lo:400:kworker.0 00000100:00080000:1.0:1789760772.507610:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000001:00080000:1.0:1789760772.512741:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973154 is committed 00002000:00010000:3.0:1789760772.515644:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23904256 -> 23908352 00010000:00010000:3.0:1789760772.515664:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23904256->23908351] (req 23904256->23908351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.515737:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23904256->23908351] (req 23904256->23908351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.515747:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23904256->23908351] (req 23904256->23908351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.515755:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.515761:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23904256->23908351] (req 23904256->23908351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.515769:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23904256->23908351] (req 23904256->23908351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.515789:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23904256->23908351] (req 23904256->23908351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.515797:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209951 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23904256->23908351] (req 23904256->23908351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.515943:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484315648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65301us (65521us total) trans 4294973154 rc 0/0 00000100:00100000:3.0:1789760772.515954:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5865 00000100:00100000:3.0:1789760772.515965:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761733192298 time 761781557183 state 0x2 ref 0 00000100:00100000:2.0:1789760772.525783:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760772.525889:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484315776 00000100:00100000:1.0:1789760772.525899:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484315776 00000100:00100000:1.0:1789760772.525958:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761733192298 time 761781557183 state 0x0 ref 1 00000100:00100000:1.0:1789760772.525973:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761781817599 time 761781557183 state 0x1 ref 1 00000100:00100000:1.0:1789760772.526005:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761781817599 time 761856935719 state 0x0 ref 1 00000100:00100000:1.0:1789760772.526017:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484315776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760772.526115:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23908352, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:1.0:1789760772.526124:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.526140:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:1.0:1789760772.526145:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760772.526152:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23908352->23912447] (req 23908352->23912447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.526168:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23908352->23912447] (req 23908352->23912447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760772.576369:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973155 is committed 00002000:00010000:0.0:1789760772.576668:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23908352 -> 23912448 00010000:00010000:0.0:1789760772.576694:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23908352->23912447] (req 23908352->23912447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.576706:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23908352->23912447] (req 23908352->23912447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.576715:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23908352->23912447] (req 23908352->23912447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.576723:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760772.576736:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23908352->23912447] (req 23908352->23912447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.576746:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23908352->23912447] (req 23908352->23912447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.576766:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23908352->23912447] (req 23908352->23912447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.576774:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209958 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23908352->23912447] (req 23908352->23912447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760772.576867:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484315776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50848us (51089us total) trans 4294973155 rc 0/0 00000100:00100000:0.0:1789760772.576877:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5866 00000100:00100000:0.0:1789760772.576894:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761781817599 time 761856935719 state 0x2 ref 0 00000100:00100000:0.0:1789760772.585421:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760772.585596:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484315904 00000100:00100000:1.0:1789760772.585607:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484315904 00000100:00100000:1.0:1789760772.585662:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761781817599 time 761856935719 state 0x0 ref 1 00000100:00100000:1.0:1789760772.585678:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761857196135 time 761856935719 state 0x1 ref 1 00000100:00100000:1.0:1789760772.585704:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761857196135 time 761916635516 state 0x0 ref 1 00000100:00100000:1.0:1789760772.585718:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484315904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760772.585830:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23912448, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:1.0:1789760772.585840:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.585855:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:1.0:1789760772.585859:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760772.585865:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23912448->23916543] (req 23912448->23916543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.585878:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23912448->23916543] (req 23912448->23916543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760772.629370:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973156 is committed 00002000:00010000:3.0:1789760772.629531:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23912448 -> 23916544 00010000:00010000:3.0:1789760772.629545:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23912448->23916543] (req 23912448->23916543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.629558:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23912448->23916543] (req 23912448->23916543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.629566:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23912448->23916543] (req 23912448->23916543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.629573:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.629577:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23912448->23916543] (req 23912448->23916543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.629586:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23912448->23916543] (req 23912448->23916543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.629609:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23912448->23916543] (req 23912448->23916543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.629616:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620995f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23912448->23916543] (req 23912448->23916543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.629707:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484315904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43987us (44290us total) trans 4294973156 rc 0/0 00000100:00100000:3.0:1789760772.629717:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5867 00000100:00100000:3.0:1789760772.629819:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761857196135 time 761916635516 state 0x2 ref 0 00000100:00100000:2.0:1789760772.635196:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760772.635445:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484316032 00000100:00100000:3.0:1789760772.635455:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484316032 00000100:00100000:3.0:1789760772.635508:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761857196135 time 761916635516 state 0x0 ref 1 00000100:00100000:3.0:1789760772.635534:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761916895932 time 761916635516 state 0x1 ref 1 00000100:00100000:3.0:1789760772.635597:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761916895932 time 761966527473 state 0x0 ref 1 00000100:00100000:3.0:1789760772.635610:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484316032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760772.635655:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23916544, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:3.0:1789760772.635664:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.635694:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=761s v=(0 0) 00010000:00010000:3.0:1789760772.635700:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760772.635706:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23916544->23920639] (req 23916544->23920639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.635720:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23916544->23920639] (req 23916544->23920639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760772.690124:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973157 is committed 00002000:00010000:3.0:1789760772.690231:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23916544 -> 23920640 00010000:00010000:3.0:1789760772.690245:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23916544->23920639] (req 23916544->23920639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.690258:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23916544->23920639] (req 23916544->23920639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.690404:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23916544->23920639] (req 23916544->23920639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.690549:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.690554:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23916544->23920639] (req 23916544->23920639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.690564:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23916544->23920639] (req 23916544->23920639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.690586:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23916544->23920639] (req 23916544->23920639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.690598:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209966 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23916544->23920639] (req 23916544->23920639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.690791:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484316032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55113us (55532us total) trans 4294973157 rc 0/0 00000100:00100000:3.0:1789760772.690802:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5868 00000100:00100000:3.0:1789760772.690812:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761916895932 time 761966527473 state 0x2 ref 0 00000100:00100000:2.0:1789760772.693486:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760772.693598:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484316160 00000100:00100000:3.0:1789760772.693609:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484316160 00000100:00100000:3.0:1789760772.693668:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761916895932 time 761966527473 state 0x0 ref 1 00000100:00100000:3.0:1789760772.693684:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761966787889 time 761966527473 state 0x1 ref 1 00000100:00100000:3.0:1789760772.693753:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761966787889 time 762024683181 state 0x0 ref 1 00000100:00100000:3.0:1789760772.693767:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484316160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760772.693816:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23920640, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:3.0:1789760772.693826:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.693842:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:3.0:1789760772.693847:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760772.693853:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23920640->23924735] (req 23920640->23924735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.693873:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23920640->23924735] (req 23920640->23924735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760772.724922:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973158 is committed 00002000:00010000:0.0:1789760772.725231:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23920640 -> 23924736 00010000:00010000:0.0:1789760772.725246:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23920640->23924735] (req 23920640->23924735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.725258:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23920640->23924735] (req 23920640->23924735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.725266:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23920640->23924735] (req 23920640->23924735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.725273:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760772.725277:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23920640->23924735] (req 23920640->23924735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.725285:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23920640->23924735] (req 23920640->23924735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.725304:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23920640->23924735] (req 23920640->23924735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.725311:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620996d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23920640->23924735] (req 23920640->23924735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760772.725408:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484316160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31639us (31925us total) trans 4294973158 rc 0/0 00000100:00100000:0.0:1789760772.725417:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5869 00000100:00100000:0.0:1789760772.725428:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761966787889 time 762024683181 state 0x2 ref 0 00000100:00100000:0.0:1789760772.736433:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760772.736665:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484316288 00000100:00100000:1.0:1789760772.736678:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484316288 00000100:00100000:1.0:1789760772.736736:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 761966787889 time 762024683181 state 0x0 ref 1 00000100:00100000:1.0:1789760772.736752:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762024943597 time 762024683181 state 0x1 ref 1 00000100:00100000:1.0:1789760772.736782:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762024943597 time 762067712829 state 0x0 ref 1 00000100:00100000:1.0:1789760772.736796:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484316288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760772.736859:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23924736, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:1.0:1789760772.736869:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.736885:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:1.0:1789760772.736889:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760772.736896:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23924736->23928831] (req 23924736->23928831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.736909:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23924736->23928831] (req 23924736->23928831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760772.777308:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973159 is committed 00002000:00010000:1.0:1789760772.777469:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23924736 -> 23928832 00010000:00010000:1.0:1789760772.777484:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23924736->23928831] (req 23924736->23928831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.777502:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23924736->23928831] (req 23924736->23928831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.777511:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23924736->23928831] (req 23924736->23928831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.777518:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760772.777523:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23924736->23928831] (req 23924736->23928831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.777532:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23924736->23928831] (req 23924736->23928831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.777551:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23924736->23928831] (req 23924736->23928831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.777559:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209974 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23924736->23928831] (req 23924736->23928831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760772.777643:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484316288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40844us (41213us total) trans 4294973159 rc 0/0 00000100:00100000:1.0:1789760772.777654:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5870 00000100:00100000:1.0:1789760772.777664:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762024943597 time 762067712829 state 0x2 ref 0 00000100:00100000:3.0:1789760772.787658:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760772.787728:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484316416 00000100:00100000:2.0:1789760772.787739:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484316416 00000100:00100000:2.0:1789760772.787792:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762024943597 time 762067712829 state 0x0 ref 1 00000100:00100000:2.0:1789760772.787808:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762067973245 time 762067712829 state 0x1 ref 1 00000100:00100000:2.0:1789760772.787842:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762067973245 time 762118772614 state 0x0 ref 1 00000100:00100000:2.0:1789760772.787855:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484316416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760772.787903:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23928832, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:2.0:1789760772.787912:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.787929:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:2.0:1789760772.787933:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760772.787940:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23928832->23932927] (req 23928832->23932927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.787953:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23928832->23932927] (req 23928832->23932927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760772.836515:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973160 is committed 00002000:00010000:3.0:1789760772.836622:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23928832 -> 23932928 00010000:00010000:3.0:1789760772.836639:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23928832->23932927] (req 23928832->23932927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.836652:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23928832->23932927] (req 23928832->23932927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.836661:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23928832->23932927] (req 23928832->23932927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.836669:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.836674:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23928832->23932927] (req 23928832->23932927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.836754:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23928832->23932927] (req 23928832->23932927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.836777:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23928832->23932927] (req 23928832->23932927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.836786:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620997b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23928832->23932927] (req 23928832->23932927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.836975:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484316416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49117us (49322us total) trans 4294973160 rc 0/0 00000100:00100000:3.0:1789760772.836986:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5871 00000100:00100000:3.0:1789760772.836995:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762067973245 time 762118772614 state 0x2 ref 0 00000100:00100000:2.0:1789760772.844034:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760772.844142:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484316544 00000100:00100000:3.0:1789760772.844152:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484316544 00000100:00100000:3.0:1789760772.844209:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762067973245 time 762118772614 state 0x0 ref 1 00000100:00100000:3.0:1789760772.844225:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762119033030 time 762118772614 state 0x1 ref 1 00000100:00100000:3.0:1789760772.844256:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762119033030 time 762175187735 state 0x0 ref 1 00000100:00100000:3.0:1789760772.844270:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484316544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760772.844319:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23932928, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:3.0:1789760772.844328:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.844346:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:3.0:1789760772.844350:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760772.844357:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23932928->23937023] (req 23932928->23937023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.844370:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23932928->23937023] (req 23932928->23937023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760772.872338:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23932928 -> 23937024 00010000:00010000:1.0:1789760772.872358:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23932928->23937023] (req 23932928->23937023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.872371:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23932928->23937023] (req 23932928->23937023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.872381:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23932928->23937023] (req 23932928->23937023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.872389:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760772.872394:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23932928->23937023] (req 23932928->23937023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.872403:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23932928->23937023] (req 23932928->23937023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.872420:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23932928->23937023] (req 23932928->23937023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.872425:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209982 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23932928->23937023] (req 23932928->23937023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760772.872503:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484316544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28230us (28476us total) trans 4294973161 rc 0/0 00000100:00100000:1.0:1789760772.872512:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5872 00000100:00100000:1.0:1789760772.872523:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762119033030 time 762175187735 state 0x2 ref 0 00000001:00080000:1.0:1789760772.872560:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973161 is committed 00000100:00100000:0.0:1789760772.877972:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760772.878107:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484316672 00000100:00100000:3.0:1789760772.878117:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484316672 00000100:00100000:3.0:1789760772.878168:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762119033030 time 762175187735 state 0x0 ref 1 00000100:00100000:3.0:1789760772.878183:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762175448151 time 762175187735 state 0x1 ref 1 00000100:00100000:3.0:1789760772.878213:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762175448151 time 762209144513 state 0x0 ref 1 00000100:00100000:3.0:1789760772.878222:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484316672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760772.878268:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23937024, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:3.0:1789760772.878275:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.878287:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:3.0:1789760772.878290:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760772.878295:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23937024->23941119] (req 23937024->23941119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.878305:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23937024->23941119] (req 23937024->23941119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760772.910251:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973162 is committed 00002000:00010000:3.0:1789760772.910327:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23937024 -> 23941120 00010000:00010000:3.0:1789760772.910342:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23937024->23941119] (req 23937024->23941119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.910360:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23937024->23941119] (req 23937024->23941119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.910369:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23937024->23941119] (req 23937024->23941119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.910376:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760772.910380:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23937024->23941119] (req 23937024->23941119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.910389:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23937024->23941119] (req 23937024->23941119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.910408:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23937024->23941119] (req 23937024->23941119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.910415:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209989 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23937024->23941119] (req 23937024->23941119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760772.911048:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484316672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32818us (33077us total) trans 4294973162 rc 0/0 00000100:00100000:3.0:1789760772.911059:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5873 00000100:00100000:3.0:1789760772.911111:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762175448151 time 762209144513 state 0x2 ref 0 00000100:00100000:3.0:1789760772.919741:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760772.919838:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484316800 00000100:00100000:1.0:1789760772.919848:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484316800 00000100:00100000:1.0:1789760772.919901:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762175448151 time 762209144513 state 0x0 ref 1 00000100:00100000:1.0:1789760772.919916:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762209404929 time 762209144513 state 0x1 ref 1 00000100:00100000:1.0:1789760772.919947:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762209404929 time 762250878088 state 0x0 ref 1 00000100:00100000:1.0:1789760772.919959:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484316800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760772.920007:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23941120, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:1.0:1789760772.920015:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.920030:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:1.0:1789760772.920034:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760772.920039:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23941120->23945215] (req 23941120->23945215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.920053:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23941120->23945215] (req 23941120->23945215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760772.947556:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973163 is committed 00002000:00010000:0.0:1789760772.947665:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23941120 -> 23945216 00010000:00010000:0.0:1789760772.947681:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23941120->23945215] (req 23941120->23945215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.947695:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23941120->23945215] (req 23941120->23945215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.947704:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23941120->23945215] (req 23941120->23945215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.947712:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760772.947718:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23941120->23945215] (req 23941120->23945215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.947727:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23941120->23945215] (req 23941120->23945215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.947746:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23941120->23945215] (req 23941120->23945215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760772.947755:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209990 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23941120->23945215] (req 23941120->23945215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760772.947884:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484316800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27921us (28146us total) trans 4294973163 rc 0/0 00000100:00100000:0.0:1789760772.947894:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5874 00000100:00100000:0.0:1789760772.947904:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762209404929 time 762250878088 state 0x2 ref 0 00000100:00100000:2.0:1789760772.952496:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760772.952668:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484316928 00000100:00100000:3.0:1789760772.952677:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484316928 00000100:00100000:3.0:1789760772.952731:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762209404929 time 762250878088 state 0x0 ref 1 00000100:00100000:3.0:1789760772.952745:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762251138504 time 762250878088 state 0x1 ref 1 00000100:00100000:3.0:1789760772.952811:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762251138504 time 762283740974 state 0x0 ref 1 00000100:00100000:3.0:1789760772.952825:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484316928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760772.952882:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23945216, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:3.0:1789760772.952891:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.952908:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:3.0:1789760772.952913:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760772.952919:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23945216->23949311] (req 23945216->23949311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760772.952933:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23945216->23949311] (req 23945216->23949311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760772.980379:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973164 is committed 00002000:00010000:2.0:1789760772.980488:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23945216 -> 23949312 00010000:00010000:2.0:1789760772.980503:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23945216->23949311] (req 23945216->23949311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.980516:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23945216->23949311] (req 23945216->23949311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.980525:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23945216->23949311] (req 23945216->23949311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.980533:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760772.980537:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23945216->23949311] (req 23945216->23949311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.980551:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23945216->23949311] (req 23945216->23949311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.980570:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23945216->23949311] (req 23945216->23949311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760772.980578:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209997 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23945216->23949311] (req 23945216->23949311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760772.980737:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484316928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27911us (28247us total) trans 4294973164 rc 0/0 00000100:00100000:2.0:1789760772.980747:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5875 00000100:00100000:2.0:1789760772.980757:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762251138504 time 762283740974 state 0x2 ref 0 00000100:00100000:3.0:1789760772.984882:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760772.985006:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484317056 00000100:00100000:1.0:1789760772.985016:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484317056 00000100:00100000:1.0:1789760772.985117:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762251138504 time 762283740974 state 0x0 ref 1 00000100:00100000:1.0:1789760772.985133:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762284001390 time 762283740974 state 0x1 ref 1 00000100:00100000:1.0:1789760772.985154:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762284001390 time 762316085458 state 0x0 ref 1 00000100:00100000:1.0:1789760772.985167:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484317056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760772.985230:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23949312, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:1.0:1789760772.985238:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.985504:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:1.0:1789760772.985509:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760772.985515:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23949312->23953407] (req 23949312->23953407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760772.985530:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23949312->23953407] (req 23949312->23953407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.004887:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:0.0:1789760773.016166:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23949312 -> 23953408 00010000:00010000:0.0:1789760773.016186:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23949312->23953407] (req 23949312->23953407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.016199:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23949312->23953407] (req 23949312->23953407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.016209:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23949312->23953407] (req 23949312->23953407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.016217:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760773.016222:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23949312->23953407] (req 23949312->23953407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.016231:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23949312->23953407] (req 23949312->23953407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.016252:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23949312->23953407] (req 23949312->23953407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.016417:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620999e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23949312->23953407] (req 23949312->23953407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760773.016556:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484317056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31386us (31684us total) trans 4294973165 rc 0/0 00000100:00100000:0.0:1789760773.016567:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5876 00000100:00100000:0.0:1789760773.016578:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762284001390 time 762316085458 state 0x2 ref 0 00000001:00080000:0.0:1789760773.016616:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973165 is committed 00000100:00100000:3.0:1789760773.026514:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760773.026653:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484317184 00000100:00100000:0.0:1789760773.026663:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484317184 00000100:00100000:0.0:1789760773.026724:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762284001390 time 762316085458 state 0x0 ref 1 00000100:00100000:0.0:1789760773.026743:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762316345874 time 762316085458 state 0x1 ref 1 00000100:00100000:0.0:1789760773.026775:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762316345874 time 762357705871 state 0x0 ref 1 00000100:00100000:0.0:1789760773.026789:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484317184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760773.026836:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23953408, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:0.0:1789760773.026845:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.026869:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:0.0:1789760773.026874:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760773.026880:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23953408->23957503] (req 23953408->23957503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.026893:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23953408->23957503] (req 23953408->23957503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760773.071718:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973166 is committed 00002000:00010000:1.0:1789760773.072011:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23953408 -> 23957504 00010000:00010000:1.0:1789760773.072026:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23953408->23957503] (req 23953408->23957503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.072039:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23953408->23957503] (req 23953408->23957503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.072047:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23953408->23957503] (req 23953408->23957503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.072054:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760773.072060:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23953408->23957503] (req 23953408->23957503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.072094:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23953408->23957503] (req 23953408->23957503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.072114:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23953408->23957503] (req 23953408->23957503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.072121:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf38263962099a5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23953408->23957503] (req 23953408->23957503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760773.072476:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484317184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45685us (45967us total) trans 4294973166 rc 0/0 00000100:00100000:1.0:1789760773.072486:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5877 00000100:00100000:1.0:1789760773.072496:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762316345874 time 762357705871 state 0x2 ref 0 00000100:00100000:2.0:1789760773.075306:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760773.075483:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484317312 00000100:00100000:0.0:1789760773.075492:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484317312 00000100:00100000:0.0:1789760773.075548:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762316345874 time 762357705871 state 0x0 ref 1 00000100:00100000:0.0:1789760773.075564:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762357966287 time 762357705871 state 0x1 ref 1 00000100:00100000:0.0:1789760773.075591:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762357966287 time 762406522293 state 0x0 ref 1 00000100:00100000:0.0:1789760773.075605:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484317312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760773.075654:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23957504, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:0.0:1789760773.075663:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.075682:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:0.0:1789760773.075687:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760773.075693:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23957504->23961599] (req 23957504->23961599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.075707:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23957504->23961599] (req 23957504->23961599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760773.113007:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973167 is committed 00002000:00010000:2.0:1789760773.113138:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23957504 -> 23961600 00010000:00010000:2.0:1789760773.113153:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23957504->23961599] (req 23957504->23961599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.113166:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23957504->23961599] (req 23957504->23961599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.113175:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23957504->23961599] (req 23957504->23961599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.113187:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760773.113192:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23957504->23961599] (req 23957504->23961599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.113201:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23957504->23961599] (req 23957504->23961599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.113220:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23957504->23961599] (req 23957504->23961599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.113227:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf38263962099ac lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23957504->23961599] (req 23957504->23961599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760773.113324:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484317312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37718us (38023us total) trans 4294973167 rc 0/0 00000100:00100000:2.0:1789760773.113333:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5878 00000100:00100000:2.0:1789760773.113343:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762357966287 time 762406522293 state 0x2 ref 0 00000100:00100000:3.0:1789760773.118360:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760773.118525:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484317440 00000100:00100000:2.0:1789760773.118536:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484317440 00000100:00100000:2.0:1789760773.118592:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762357966287 time 762406522293 state 0x0 ref 1 00000100:00100000:2.0:1789760773.118607:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762406782709 time 762406522293 state 0x1 ref 1 00000100:00100000:2.0:1789760773.118691:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762406782709 time 762449621545 state 0x0 ref 1 00000100:00100000:2.0:1789760773.118707:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484317440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760773.118755:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23961600, mtime: 0x6aad9504, atime: 0x0, ctime: 0x6aad9504, blocks: 0xb807 00010000:00010000:2.0:1789760773.118764:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.118781:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:2.0:1789760773.118786:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760773.118796:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23961600->23965695] (req 23961600->23965695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.118811:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23961600->23965695] (req 23961600->23965695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760773.151978:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23961600 -> 23965696 00002000:00010000:2.0:1789760773.151986:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760772 -> 1789760773 00002000:00010000:2.0:1789760773.151989:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760772 -> 1789760773 00010000:00010000:2.0:1789760773.152001:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23961600->23965695] (req 23961600->23965695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.152014:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23961600->23965695] (req 23961600->23965695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.152023:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23961600->23965695] (req 23961600->23965695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.152030:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760773.152035:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23961600->23965695] (req 23961600->23965695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.152043:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23961600->23965695] (req 23961600->23965695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.152061:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23961600->23965695] (req 23961600->23965695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.152103:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099b3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23961600->23965695] (req 23961600->23965695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760773.152182:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484317440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33473us (33827us total) trans 4294973168 rc 0/0 00000100:00100000:2.0:1789760773.152191:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5879 00000100:00100000:2.0:1789760773.152200:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762406782709 time 762449621545 state 0x2 ref 0 00000001:00080000:0.0:1789760773.152334:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973168 is committed 00000100:00100000:0.0:1789760773.157692:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760773.157989:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484317568 00000100:00100000:1.0:1789760773.157999:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484317568 00000100:00100000:1.0:1789760773.158054:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762406782709 time 762449621545 state 0x0 ref 1 00000100:00100000:1.0:1789760773.158103:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762449881961 time 762449621545 state 0x1 ref 1 00000100:00100000:1.0:1789760773.158137:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762449881961 time 762489067937 state 0x0 ref 1 00000100:00100000:1.0:1789760773.158150:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484317568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760773.158199:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23965696, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760773.158208:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.158224:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:1.0:1789760773.158228:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760773.158235:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23965696->23969791] (req 23965696->23969791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.158249:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23965696->23969791] (req 23965696->23969791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760773.190442:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973169 is committed 00002000:00010000:1.0:1789760773.190608:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23965696 -> 23969792 00010000:00010000:1.0:1789760773.190623:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23965696->23969791] (req 23965696->23969791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.190636:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23965696->23969791] (req 23965696->23969791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.190645:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23965696->23969791] (req 23965696->23969791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.190652:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760773.190657:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23965696->23969791] (req 23965696->23969791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.190666:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23965696->23969791] (req 23965696->23969791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.190691:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23965696->23969791] (req 23965696->23969791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.190699:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099ba lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23965696->23969791] (req 23965696->23969791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760773.190782:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484317568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32631us (33104us total) trans 4294973169 rc 0/0 00000100:00100000:1.0:1789760773.190792:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5880 00000100:00100000:1.0:1789760773.190800:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762449881961 time 762489067937 state 0x2 ref 0 00000100:00100000:1.0:1789760773.194204:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760773.194290:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484317696 00000100:00100000:0.0:1789760773.194299:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484317696 00000100:00100000:0.0:1789760773.194355:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762449881961 time 762489067937 state 0x0 ref 1 00000100:00100000:0.0:1789760773.194368:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762489328353 time 762489067937 state 0x1 ref 1 00000100:00100000:0.0:1789760773.194437:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762489328353 time 762525367035 state 0x0 ref 1 00000100:00100000:0.0:1789760773.194451:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484317696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760773.194502:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23969792, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:0.0:1789760773.194517:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.194532:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:0.0:1789760773.194537:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760773.194543:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23969792->23973887] (req 23969792->23973887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.194557:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23969792->23973887] (req 23969792->23973887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760773.223898:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973170 is committed 00002000:00010000:0.0:1789760773.224053:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23969792 -> 23973888 00010000:00010000:0.0:1789760773.224098:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23969792->23973887] (req 23969792->23973887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.224112:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23969792->23973887] (req 23969792->23973887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.224121:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23969792->23973887] (req 23969792->23973887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.224129:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760773.224134:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23969792->23973887] (req 23969792->23973887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.224144:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23969792->23973887] (req 23969792->23973887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.224168:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23969792->23973887] (req 23969792->23973887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.224176:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf38263962099c1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23969792->23973887] (req 23969792->23973887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760773.224259:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484317696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29807us (30060us total) trans 4294973170 rc 0/0 00000100:00100000:0.0:1789760773.224269:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5881 00000100:00100000:0.0:1789760773.224279:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762489328353 time 762525367035 state 0x2 ref 0 00000100:00100000:0.0:1789760773.230247:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760773.230336:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484317824 00000100:00100000:2.0:1789760773.230353:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484317824 00000100:00100000:2.0:1789760773.230410:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762489328353 time 762525367035 state 0x0 ref 1 00000100:00100000:2.0:1789760773.230424:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762525627451 time 762525367035 state 0x1 ref 1 00000100:00100000:2.0:1789760773.230455:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762525627451 time 762561386128 state 0x0 ref 1 00000100:00100000:2.0:1789760773.230468:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484317824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760773.230513:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23973888, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:2.0:1789760773.230523:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.230538:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:2.0:1789760773.230542:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760773.230548:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23973888->23977983] (req 23973888->23977983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.230561:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23973888->23977983] (req 23973888->23977983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760773.270371:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973171 is committed 00002000:00010000:1.0:1789760773.270516:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23973888 -> 23977984 00010000:00010000:1.0:1789760773.270532:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23973888->23977983] (req 23973888->23977983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.270545:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23973888->23977983] (req 23973888->23977983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.270554:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23973888->23977983] (req 23973888->23977983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.270560:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760773.270565:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23973888->23977983] (req 23973888->23977983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.270579:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23973888->23977983] (req 23973888->23977983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.270598:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23973888->23977983] (req 23973888->23977983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.270605:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf38263962099c8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23973888->23977983] (req 23973888->23977983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760773.270735:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484317824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40264us (40490us total) trans 4294973171 rc 0/0 00000100:00100000:1.0:1789760773.270746:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5882 00000100:00100000:1.0:1789760773.270757:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762525627451 time 762561386128 state 0x2 ref 0 00000100:00100000:3.0:1789760773.273627:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760773.275870:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484317952 00000100:00100000:2.0:1789760773.275885:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484317952 00000100:00100000:2.0:1789760773.276023:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762525627451 time 762561386128 state 0x0 ref 1 00000100:00100000:2.0:1789760773.276040:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762561646544 time 762561386128 state 0x1 ref 1 00000100:00100000:2.0:1789760773.276175:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762561646544 time 762607104024 state 0x0 ref 1 00000100:00100000:2.0:1789760773.276189:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484317952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760773.276251:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23977984, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:2.0:1789760773.276260:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.276275:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:2.0:1789760773.276280:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760773.276286:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23977984->23982079] (req 23977984->23982079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.276301:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23977984->23982079] (req 23977984->23982079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760773.310477:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973172 is committed 00002000:00010000:2.0:1789760773.310583:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23977984 -> 23982080 00010000:00010000:2.0:1789760773.310598:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23977984->23982079] (req 23977984->23982079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.310610:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23977984->23982079] (req 23977984->23982079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.310618:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23977984->23982079] (req 23977984->23982079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.310626:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760773.310631:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23977984->23982079] (req 23977984->23982079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.310641:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23977984->23982079] (req 23977984->23982079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.310658:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23977984->23982079] (req 23977984->23982079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.310663:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099cf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23977984->23982079] (req 23977984->23982079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760773.310829:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484317952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34638us (37206us total) trans 4294973172 rc 0/0 00000100:00100000:2.0:1789760773.310839:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5883 00000100:00100000:2.0:1789760773.310848:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762561646544 time 762607104024 state 0x2 ref 0 00000100:00100000:3.0:1789760773.315802:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760773.316168:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484318080 00000100:00100000:1.0:1789760773.316178:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484318080 00000100:00100000:1.0:1789760773.317133:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762561646544 time 762607104024 state 0x0 ref 1 00000100:00100000:1.0:1789760773.317158:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762607364440 time 762607104024 state 0x1 ref 1 00000100:00100000:1.0:1789760773.317204:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762607364440 time 762648134401 state 0x0 ref 1 00000100:00100000:1.0:1789760773.317221:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484318080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760773.317275:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23982080, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760773.317285:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.317299:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:1.0:1789760773.317304:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760773.317311:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23982080->23986175] (req 23982080->23986175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.317325:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23982080->23986175] (req 23982080->23986175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760773.346887:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973173 is committed 00002000:00010000:0.0:1789760773.347724:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23982080 -> 23986176 00010000:00010000:0.0:1789760773.347741:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23982080->23986175] (req 23982080->23986175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.347755:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23982080->23986175] (req 23982080->23986175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.347765:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23982080->23986175] (req 23982080->23986175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.347773:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760773.347778:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23982080->23986175] (req 23982080->23986175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.347793:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23982080->23986175] (req 23982080->23986175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.347812:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23982080->23986175] (req 23982080->23986175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.347820:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf38263962099d6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23982080->23986175] (req 23982080->23986175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760773.347923:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484318080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30704us (32126us total) trans 4294973173 rc 0/0 00000100:00100000:0.0:1789760773.347933:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5884 00000100:00100000:0.0:1789760773.347943:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762607364440 time 762648134401 state 0x2 ref 0 00000100:00100000:0.0:1789760773.349157:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760773.349191:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760773.349218:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760773.349237:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760773.349260:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760773.349283:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484318336 00000100:00100000:0.0:1789760773.349291:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484318336 00000100:00100000:0.0:1789760773.349340:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484318592 00000100:00100000:0.0:1789760773.349343:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484318592 00000100:00100000:0.0:1789760773.349366:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484318464 00000100:00100000:0.0:1789760773.349369:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484318464 00000100:00100000:0.0:1789760773.349380:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484318720 00000100:00100000:0.0:1789760773.349382:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484318720 00000100:00100000:0.0:1789760773.349400:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 981 00000100:00100000:0.0:1789760773.349410:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484318336:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760773.349458:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 982 00000100:00100000:0.0:1789760773.349469:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484318336:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 60us (316us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760773.349472:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484318592:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760773.349484:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 981 00000100:00100000:0.0:1789760773.349503:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 983 00000100:00100000:3.0:1789760773.349513:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484318592:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 44us (319us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760773.349514:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484318464:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760773.349526:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 982 00000100:00100000:3.0:1789760773.349543:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 984 00000100:00100000:3.0:1789760773.349551:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484318720:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760773.349575:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484318720:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 26us (315us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760773.349581:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 984 00000100:00100000:0.0:1789760773.349619:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484318464:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 108us (379us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760773.349628:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 983 00000100:00100000:0.0:1789760773.349650:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484318208 00000100:00100000:0.0:1789760773.349653:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484318208 00000100:00100000:0.0:1789760773.349687:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 491 00000100:00100000:0.0:1789760773.349695:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8038450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484318208:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760773.349717:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8038450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484318208:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 24us (499us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760773.349723:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 491 00000100:00080000:0.0:1789760773.349749:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:0.0:1789760773.349753:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:3.0:1789760773.356975:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760773.357056:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484318848 00000100:00100000:0.0:1789760773.357192:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484318848 00000100:00100000:1.0:1789760773.357325:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762607364440 time 762648134401 state 0x0 ref 1 00000100:00100000:1.0:1789760773.357342:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762648394817 time 762648134401 state 0x1 ref 1 00000100:00100000:1.0:1789760773.357421:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762648394817 time 762688351187 state 0x0 ref 1 00000100:00100000:1.0:1789760773.357435:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484318848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760773.357494:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23986176, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760773.357504:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.357520:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:1.0:1789760773.357534:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760773.357540:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23986176->23990271] (req 23986176->23990271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.357561:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23986176->23990271] (req 23986176->23990271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760773.414521:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23986176 -> 23990272 00010000:00010000:3.0:1789760773.414542:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23986176->23990271] (req 23986176->23990271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.414555:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23986176->23990271] (req 23986176->23990271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.414564:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23986176->23990271] (req 23986176->23990271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.414571:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760773.414575:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23986176->23990271] (req 23986176->23990271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.414712:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23986176->23990271] (req 23986176->23990271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.414731:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23986176->23990271] (req 23986176->23990271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.414739:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf38263962099dd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23986176->23990271] (req 23986176->23990271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760773.414890:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973174 is committed 00000100:00100000:3.0:1789760773.415000:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484318848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57563us (58030us total) trans 4294973174 rc 0/0 00000100:00100000:3.0:1789760773.415009:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5885 00000100:00100000:3.0:1789760773.415017:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762648394817 time 762688351187 state 0x2 ref 0 00000100:00100000:0.0:1789760773.418361:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760773.418532:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484318976 00000100:00100000:2.0:1789760773.418542:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484318976 00000100:00100000:2.0:1789760773.418601:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762648394817 time 762688351187 state 0x0 ref 1 00000100:00100000:2.0:1789760773.418617:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762688611603 time 762688351187 state 0x1 ref 1 00000100:00100000:2.0:1789760773.418646:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762688611603 time 762749577729 state 0x0 ref 1 00000100:00100000:2.0:1789760773.418659:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484318976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760773.418707:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23990272, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:2.0:1789760773.418716:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.418732:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:2.0:1789760773.418737:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760773.418743:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23990272->23994367] (req 23990272->23994367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760773.418757:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23990272->23994367] (req 23990272->23994367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760773.470726:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973175 is committed 00002000:00010000:1.0:1789760773.471148:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23990272 -> 23994368 00010000:00010000:1.0:1789760773.471275:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23990272->23994367] (req 23990272->23994367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.471293:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23990272->23994367] (req 23990272->23994367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.471302:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23990272->23994367] (req 23990272->23994367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.471309:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760773.471314:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23990272->23994367] (req 23990272->23994367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.471322:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23990272->23994367] (req 23990272->23994367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.471343:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23990272->23994367] (req 23990272->23994367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.471351:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf38263962099e4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23990272->23994367] (req 23990272->23994367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760773.471664:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484318976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53002us (53308us total) trans 4294973175 rc 0/0 00000100:00100000:2.0:1789760773.471675:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5886 00000100:00100000:2.0:1789760773.471685:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762688611603 time 762749577729 state 0x2 ref 0 00000100:00100000:2.0:1789760773.475627:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760773.475717:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484319104 00000100:00100000:3.0:1789760773.475734:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484319104 00000100:00100000:3.0:1789760773.475817:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762688611603 time 762749577729 state 0x0 ref 1 00000100:00100000:3.0:1789760773.475838:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762749838145 time 762749577729 state 0x1 ref 1 00000100:00100000:3.0:1789760773.475910:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762749838145 time 762806839782 state 0x0 ref 1 00000100:00100000:3.0:1789760773.475926:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484319104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760773.475998:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23994368, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:3.0:1789760773.476009:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.476028:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:3.0:1789760773.476033:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760773.476041:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23994368->23998463] (req 23994368->23998463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.476058:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23994368->23998463] (req 23994368->23998463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760773.516915:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23994368 -> 23998464 00010000:00010000:0.0:1789760773.516945:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23994368->23998463] (req 23994368->23998463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.516979:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23994368->23998463] (req 23994368->23998463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.516990:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23994368->23998463] (req 23994368->23998463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.516998:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760773.517004:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23994368->23998463] (req 23994368->23998463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.517015:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23994368->23998463] (req 23994368->23998463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.517040:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23994368->23998463] (req 23994368->23998463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.517049:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf38263962099eb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23994368->23998463] (req 23994368->23998463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760773.517476:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484319104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41546us (41850us total) trans 4294973176 rc 0/0 00000100:00100000:0.0:1789760773.517490:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5887 00000100:00100000:0.0:1789760773.517503:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762749838145 time 762806839782 state 0x2 ref 0 00000001:00080000:1.0:1789760773.517615:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973176 is committed 00000100:00100000:0.0:1789760773.521096:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760773.521253:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484319232 00000100:00100000:1.0:1789760773.521266:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484319232 00000100:00100000:1.0:1789760773.521341:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762749838145 time 762806839782 state 0x0 ref 1 00000100:00100000:1.0:1789760773.521362:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762807100198 time 762806839782 state 0x1 ref 1 00000100:00100000:1.0:1789760773.521400:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762807100198 time 762852330636 state 0x0 ref 1 00000100:00100000:1.0:1789760773.521415:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484319232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760773.521481:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 23998464, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760773.521491:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.521511:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:1.0:1789760773.521516:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760773.521524:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23998464->24002559] (req 23998464->24002559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.521540:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23998464->24002559] (req 23998464->24002559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760773.566804:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973177 is committed 00002000:00010000:0.0:1789760773.566974:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 23998464 -> 24002560 00010000:00010000:0.0:1789760773.566995:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23998464->24002559] (req 23998464->24002559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.567010:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23998464->24002559] (req 23998464->24002559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.567020:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23998464->24002559] (req 23998464->24002559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.567027:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760773.567033:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23998464->24002559] (req 23998464->24002559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.567042:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23998464->24002559] (req 23998464->24002559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.567225:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23998464->24002559] (req 23998464->24002559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.567235:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf38263962099f2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [23998464->24002559] (req 23998464->24002559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760773.567675:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484319232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46253us (46613us total) trans 4294973177 rc 0/0 00000100:00100000:1.0:1789760773.567690:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5888 00000100:00100000:1.0:1789760773.567704:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762807100198 time 762852330636 state 0x2 ref 0 00000100:00100000:2.0:1789760773.571940:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760773.572047:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484319360 00000100:00100000:1.0:1789760773.572060:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484319360 00000100:00100000:1.0:1789760773.572454:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762807100198 time 762852330636 state 0x0 ref 1 00000100:00100000:1.0:1789760773.572474:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762852591052 time 762852330636 state 0x1 ref 1 00000100:00100000:1.0:1789760773.572659:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762852591052 time 762903587410 state 0x0 ref 1 00000100:00100000:1.0:1789760773.572675:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484319360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760773.572743:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24002560, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760773.572754:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.572775:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:1.0:1789760773.572780:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760773.572788:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24002560->24006655] (req 24002560->24006655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.572804:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24002560->24006655] (req 24002560->24006655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760773.621782:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973178 is committed 00002000:00010000:3.0:1789760773.621954:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24002560 -> 24006656 00010000:00010000:3.0:1789760773.621976:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24002560->24006655] (req 24002560->24006655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.621993:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24002560->24006655] (req 24002560->24006655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.622004:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24002560->24006655] (req 24002560->24006655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.622013:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760773.622019:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24002560->24006655] (req 24002560->24006655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.622029:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24002560->24006655] (req 24002560->24006655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.622057:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24002560->24006655] (req 24002560->24006655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.622263:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf38263962099f9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24002560->24006655] (req 24002560->24006655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760773.622425:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484319360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49745us (50557us total) trans 4294973178 rc 0/0 00000100:00100000:3.0:1789760773.622437:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5889 00000100:00100000:3.0:1789760773.622448:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762852591052 time 762903587410 state 0x2 ref 0 00000100:00100000:2.0:1789760773.627246:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760773.627622:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484319488 00000100:00100000:3.0:1789760773.627635:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484319488 00000100:00100000:3.0:1789760773.627707:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762852591052 time 762903587410 state 0x0 ref 1 00000100:00100000:3.0:1789760773.627725:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762903847826 time 762903587410 state 0x1 ref 1 00000100:00100000:3.0:1789760773.627759:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762903847826 time 762958690250 state 0x0 ref 1 00000100:00100000:3.0:1789760773.627775:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484319488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760773.627839:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24006656, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:3.0:1789760773.627850:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.627885:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=762s v=(0 0) 00010000:00010000:3.0:1789760773.627891:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760773.627899:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24006656->24010751] (req 24006656->24010751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.627921:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24006656->24010751] (req 24006656->24010751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760773.669581:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24006656 -> 24010752 00000001:00080000:1.0:1789760773.669594:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973179 is committed 00010000:00010000:0.0:1789760773.669673:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24006656->24010751] (req 24006656->24010751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.669697:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24006656->24010751] (req 24006656->24010751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.669707:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24006656->24010751] (req 24006656->24010751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.669716:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760773.669722:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24006656->24010751] (req 24006656->24010751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.669732:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24006656->24010751] (req 24006656->24010751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.669758:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24006656->24010751] (req 24006656->24010751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.669766:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a00 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24006656->24010751] (req 24006656->24010751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760773.669991:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484319488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42212us (42753us total) trans 4294973179 rc 0/0 00000100:00100000:0.0:1789760773.670005:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5890 00000100:00100000:0.0:1789760773.670018:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762903847826 time 762958690250 state 0x2 ref 0 00000100:00100000:0.0:1789760773.672748:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760773.672923:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484319616 00000100:00100000:3.0:1789760773.672937:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484319616 00000100:00100000:3.0:1789760773.673015:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762903847826 time 762958690250 state 0x0 ref 1 00000100:00100000:3.0:1789760773.673035:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762958950666 time 762958690250 state 0x1 ref 1 00000100:00100000:1.0:1789760773.673292:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762958950666 time 763004218126 state 0x0 ref 1 00000100:00100000:1.0:1789760773.673311:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484319616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760773.673375:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24010752, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760773.673385:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.673404:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760773.673410:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760773.673417:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24010752->24014847] (req 24010752->24014847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.673435:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24010752->24014847] (req 24010752->24014847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760773.704488:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24010752 -> 24014848 00010000:00010000:1.0:1789760773.704521:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24010752->24014847] (req 24010752->24014847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760773.704533:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973180 is committed 00010000:00010000:1.0:1789760773.704536:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24010752->24014847] (req 24010752->24014847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.704545:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24010752->24014847] (req 24010752->24014847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.704552:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760773.704557:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24010752->24014847] (req 24010752->24014847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.704568:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24010752->24014847] (req 24010752->24014847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.704592:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24010752->24014847] (req 24010752->24014847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.704600:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a07 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24010752->24014847] (req 24010752->24014847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760773.704732:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484319616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31418us (31991us total) trans 4294973180 rc 0/0 00000100:00100000:1.0:1789760773.704744:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5891 00000100:00100000:1.0:1789760773.704756:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762958950666 time 763004218126 state 0x2 ref 0 00000100:00100000:2.0:1789760773.709873:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760773.710260:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484319744 00000100:00100000:1.0:1789760773.710274:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484319744 00000100:00100000:1.0:1789760773.710352:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 762958950666 time 763004218126 state 0x0 ref 1 00000100:00100000:1.0:1789760773.710370:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763004478542 time 763004218126 state 0x1 ref 1 00000100:00100000:1.0:1789760773.710403:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763004478542 time 763041333084 state 0x0 ref 1 00000100:00100000:1.0:1789760773.710417:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484319744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760773.710492:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24014848, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760773.710503:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.710528:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760773.710533:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760773.710541:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24014848->24018943] (req 24014848->24018943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.710556:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24014848->24018943] (req 24014848->24018943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760773.764880:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24014848 -> 24018944 00010000:00010000:0.0:1789760773.764908:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24014848->24018943] (req 24014848->24018943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.764924:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24014848->24018943] (req 24014848->24018943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.764934:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24014848->24018943] (req 24014848->24018943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.764943:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760773.764949:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24014848->24018943] (req 24014848->24018943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.764959:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24014848->24018943] (req 24014848->24018943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.764983:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24014848->24018943] (req 24014848->24018943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.764991:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209a0e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24014848->24018943] (req 24014848->24018943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760773.765152:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484319744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54730us (55292us total) trans 4294973181 rc 0/0 00000100:00100000:0.0:1789760773.765165:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5892 00000100:00100000:0.0:1789760773.765177:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763004478542 time 763041333084 state 0x2 ref 0 00000001:00080000:3.0:1789760773.765616:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973181 is committed 00000100:00100000:2.0:1789760773.770246:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760773.770438:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484319872 00000100:00100000:3.0:1789760773.770451:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484319872 00000100:00100000:3.0:1789760773.770524:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763004478542 time 763041333084 state 0x0 ref 1 00000100:00100000:3.0:1789760773.770545:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763041593500 time 763041333084 state 0x1 ref 1 00000100:00100000:3.0:1789760773.770747:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763041593500 time 763101675200 state 0x0 ref 1 00000100:00100000:3.0:1789760773.770763:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484319872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760773.770832:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24018944, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:3.0:1789760773.770843:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.770862:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:3.0:1789760773.770867:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760773.770874:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24018944->24023039] (req 24018944->24023039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.770892:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24018944->24023039] (req 24018944->24023039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760773.802945:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973182 is committed 00002000:00010000:3.0:1789760773.803105:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24018944 -> 24023040 00010000:00010000:3.0:1789760773.803125:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24018944->24023039] (req 24018944->24023039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.803141:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24018944->24023039] (req 24018944->24023039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.803151:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24018944->24023039] (req 24018944->24023039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.803160:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760773.803166:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24018944->24023039] (req 24018944->24023039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.803177:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24018944->24023039] (req 24018944->24023039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.803202:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24018944->24023039] (req 24018944->24023039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.803210:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a15 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24018944->24023039] (req 24018944->24023039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760773.803332:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484319872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32566us (33097us total) trans 4294973182 rc 0/0 00000100:00100000:3.0:1789760773.803344:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5893 00000100:00100000:3.0:1789760773.803356:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763041593500 time 763101675200 state 0x2 ref 0 00000100:00100000:0.0:1789760773.807719:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760773.807829:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484320000 00000100:00100000:3.0:1789760773.807843:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484320000 00000100:00100000:3.0:1789760773.807977:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763041593500 time 763101675200 state 0x0 ref 1 00000100:00100000:3.0:1789760773.807996:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763101935616 time 763101675200 state 0x1 ref 1 00000100:00100000:3.0:1789760773.808100:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763101935616 time 763138994960 state 0x0 ref 1 00000100:00100000:3.0:1789760773.808116:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484320000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760773.808248:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24023040, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:3.0:1789760773.808259:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.808280:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:3.0:1789760773.808285:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760773.808292:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24023040->24027135] (req 24023040->24027135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.808308:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24023040->24027135] (req 24023040->24027135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760773.840794:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973183 is committed 00002000:00010000:3.0:1789760773.840812:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24023040 -> 24027136 00010000:00010000:3.0:1789760773.840831:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24023040->24027135] (req 24023040->24027135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.840847:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24023040->24027135] (req 24023040->24027135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.840857:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24023040->24027135] (req 24023040->24027135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.840865:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760773.840871:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24023040->24027135] (req 24023040->24027135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.840881:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24023040->24027135] (req 24023040->24027135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.840907:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24023040->24027135] (req 24023040->24027135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.840920:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a1c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24023040->24027135] (req 24023040->24027135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760773.841130:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484320000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32946us (33354us total) trans 4294973183 rc 0/0 00000100:00100000:3.0:1789760773.841147:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5894 00000100:00100000:3.0:1789760773.841159:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763101935616 time 763138994960 state 0x2 ref 0 00000100:00100000:2.0:1789760773.846455:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760773.846561:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484320128 00000100:00100000:1.0:1789760773.846574:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484320128 00000100:00100000:1.0:1789760773.846646:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763101935616 time 763138994960 state 0x0 ref 1 00000100:00100000:1.0:1789760773.846666:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763139255376 time 763138994960 state 0x1 ref 1 00000100:00100000:1.0:1789760773.846703:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763139255376 time 763177633161 state 0x0 ref 1 00000100:00100000:1.0:1789760773.846717:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484320128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760773.846791:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24027136, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760773.846802:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.846821:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760773.846826:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760773.846834:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24027136->24031231] (req 24027136->24031231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.846851:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24027136->24031231] (req 24027136->24031231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760773.887184:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973184 is committed 00002000:00010000:0.0:1789760773.887406:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24027136 -> 24031232 00010000:00010000:0.0:1789760773.887426:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24027136->24031231] (req 24027136->24031231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.887441:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24027136->24031231] (req 24027136->24031231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.887451:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24027136->24031231] (req 24027136->24031231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.887458:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760773.887464:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24027136->24031231] (req 24027136->24031231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.887474:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24027136->24031231] (req 24027136->24031231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.887499:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24027136->24031231] (req 24027136->24031231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.887508:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209a23 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24027136->24031231] (req 24027136->24031231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760773.887633:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484320128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40911us (41186us total) trans 4294973184 rc 0/0 00000100:00100000:0.0:1789760773.887651:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5895 00000100:00100000:0.0:1789760773.887725:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763139255376 time 763177633161 state 0x2 ref 0 00000100:00100000:0.0:1789760773.898222:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760773.898779:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484320256 00000100:00100000:1.0:1789760773.898793:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484320256 00000100:00100000:1.0:1789760773.899521:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763139255376 time 763177633161 state 0x0 ref 1 00000100:00100000:1.0:1789760773.899547:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763177893577 time 763177633161 state 0x1 ref 1 00000100:00100000:1.0:1789760773.899591:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763177893577 time 763230521631 state 0x0 ref 1 00000100:00100000:1.0:1789760773.899609:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706dc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484320256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760773.899685:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24031232, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760773.899696:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.899716:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760773.899722:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760773.899730:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24031232->24035327] (req 24031232->24035327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.899747:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24031232->24035327] (req 24031232->24035327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760773.939481:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973185 is committed 00002000:00010000:0.0:1789760773.939583:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24031232 -> 24035328 00010000:00010000:0.0:1789760773.939604:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24031232->24035327] (req 24031232->24035327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.939619:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24031232->24035327] (req 24031232->24035327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.939629:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24031232->24035327] (req 24031232->24035327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.939637:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760773.939643:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24031232->24035327] (req 24031232->24035327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.939654:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24031232->24035327] (req 24031232->24035327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.939684:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24031232->24035327] (req 24031232->24035327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760773.939693:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209a2a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24031232->24035327] (req 24031232->24035327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760773.939820:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706dc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484320256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40211us (41606us total) trans 4294973185 rc 0/0 00000100:00100000:0.0:1789760773.939832:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5896 00000100:00100000:0.0:1789760773.939842:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763177893577 time 763230521631 state 0x2 ref 0 00000100:00100000:1.0:1789760773.945592:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760773.945882:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484320384 00000100:00100000:3.0:1789760773.945896:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484320384 00000100:00100000:3.0:1789760773.946447:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763177893577 time 763230521631 state 0x0 ref 1 00000100:00100000:3.0:1789760773.946468:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763230782047 time 763230521631 state 0x1 ref 1 00000100:00100000:3.0:1789760773.946498:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763230782047 time 763277428629 state 0x0 ref 1 00000100:00100000:3.0:1789760773.946515:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9651c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484320384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760773.946585:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24035328, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:3.0:1789760773.946597:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.946618:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:3.0:1789760773.946623:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760773.946632:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24035328->24039423] (req 24035328->24039423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760773.946649:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24035328->24039423] (req 24035328->24039423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760773.995153:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24035328 -> 24039424 00010000:00010000:1.0:1789760773.995350:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24035328->24039423] (req 24035328->24039423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.995366:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24035328->24039423] (req 24035328->24039423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.995376:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24035328->24039423] (req 24035328->24039423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.995385:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760773.995390:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24035328->24039423] (req 24035328->24039423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.995400:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24035328->24039423] (req 24035328->24039423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.995425:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24035328->24039423] (req 24035328->24039423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760773.995434:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209a31 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24035328->24039423] (req 24035328->24039423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760773.996516:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9651c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484320384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49993us (50926us total) trans 4294973186 rc 0/0 00000100:00100000:1.0:1789760773.996531:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5897 00000100:00100000:1.0:1789760773.996544:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763230782047 time 763277428629 state 0x2 ref 0 00000001:00080000:1.0:1789760773.996621:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973186 is committed 00000100:00100000:2.0:1789760774.000259:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760774.000605:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484320512 00000100:00100000:3.0:1789760774.000619:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484320512 00000100:00100000:3.0:1789760774.000764:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763230782047 time 763277428629 state 0x0 ref 1 00000100:00100000:3.0:1789760774.000784:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763277689045 time 763277428629 state 0x1 ref 1 00000100:00100000:3.0:1789760774.000824:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763277689045 time 763331754606 state 0x0 ref 1 00000100:00100000:3.0:1789760774.000839:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484320512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760774.000916:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24039424, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:3.0:1789760774.000927:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.000948:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:3.0:1789760774.000954:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760774.000962:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24039424->24043519] (req 24039424->24043519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.000979:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24039424->24043519] (req 24039424->24043519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1789760774.001459:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760774.028332:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5115500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508936064:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760774.028425:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760774.028711:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:2.0:1789760774.028723:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb9f3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508936192:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1789760774.028762:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760774.028800:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508936064 00000100:00100000:2.0:1789760774.028808:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508936064 00000100:00100000:2.0:1789760774.028879:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508936192 00000100:00100000:2.0:1789760774.028882:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508936192 00000100:00100000:2.0:1789760774.028899:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 406 00000100:00100000:2.0:1789760774.028908:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508936064:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:2.0:1789760774.028936:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527892480 pending 1703936 free 3834642432 avail 3825205248 00000100:00100000:2.0:1789760774.028976:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508936064:12345-0@lo:13:osp-pre-0-1.0 Request processed in 70us (551us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760774.029136:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 406 00000100:00100000:2.0:1789760774.029161:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 407 00000100:00100000:2.0:1789760774.029167:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508936192:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:2.0:1789760774.029179:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527892480 pending 1703936 free 3834642432 avail 3825205248 00000100:00100000:2.0:1789760774.029207:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508936192:12345-0@lo:13:osp-pre-0-0.0 Request processed in 41us (445us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760774.029213:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 407 00010000:00010000:3.0:1789760774.029462:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760774.030015:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb9f3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508936192:0@lo:13:osp-pre-0-0.0 00000100:00080000:2.0:1789760774.030046:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760774.036442:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1789760774.036468:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5117b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508936320:0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760774.036663:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760774.037477:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508936320 00000100:00100000:0.0:1789760774.037489:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508936320 00000100:00100000:0.0:1789760774.037577:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 465 00000100:00100000:0.0:1789760774.037593:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508936320:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:0.0:1789760774.037647:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:0.0:1789760774.037721:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508936320:12345-0@lo:41:osp-pre-0-1.0 Request processed in 134us (1056us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760774.037729:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 465 00000100:00100000:0.0:1789760774.037791:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5117b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508936320:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760774.037836:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5115500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508936064:0@lo:13:osp-pre-0-1.0 00002000:00010000:0.0:1789760774.052014:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24039424 -> 24043520 00010000:00010000:0.0:1789760774.052042:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24039424->24043519] (req 24039424->24043519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.052098:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24039424->24043519] (req 24039424->24043519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.052116:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24039424->24043519] (req 24039424->24043519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.052124:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760774.052130:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24039424->24043519] (req 24039424->24043519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.052141:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24039424->24043519] (req 24039424->24043519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.052164:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24039424->24043519] (req 24039424->24043519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.052172:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209a38 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24039424->24043519] (req 24039424->24043519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760774.052281:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484320512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51439us (52030us total) trans 4294973187 rc 0/0 00000100:00100000:0.0:1789760774.052293:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5898 00000100:00100000:0.0:1789760774.052305:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763277689045 time 763331754606 state 0x2 ref 0 00000001:00080000:1.0:1789760774.052443:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973187 is committed 00000100:00100000:0.0:1789760774.055594:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760774.055691:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484320640 00000100:00100000:1.0:1789760774.055710:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484320640 00000100:00100000:1.0:1789760774.055777:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763277689045 time 763331754606 state 0x0 ref 1 00000100:00100000:1.0:1789760774.055794:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763332015022 time 763331754606 state 0x1 ref 1 00000100:00100000:1.0:1789760774.055829:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763332015022 time 763386759709 state 0x0 ref 1 00000100:00100000:1.0:1789760774.055842:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484320640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760774.055910:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24043520, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760774.055923:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.055943:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760774.055948:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760774.055955:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24043520->24047615] (req 24043520->24047615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.055972:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24043520->24047615] (req 24043520->24047615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760774.090189:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973188 is committed 00002000:00010000:2.0:1789760774.090460:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24043520 -> 24047616 00010000:00010000:2.0:1789760774.090480:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24043520->24047615] (req 24043520->24047615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.090497:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24043520->24047615] (req 24043520->24047615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.090507:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24043520->24047615] (req 24043520->24047615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.090515:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760774.090520:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24043520->24047615] (req 24043520->24047615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.090530:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24043520->24047615] (req 24043520->24047615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.090554:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24043520->24047615] (req 24043520->24047615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.090563:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209a3f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24043520->24047615] (req 24043520->24047615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760774.090720:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484320640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34872us (35132us total) trans 4294973188 rc 0/0 00000100:00100000:2.0:1789760774.090732:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5899 00000100:00100000:2.0:1789760774.090744:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763332015022 time 763386759709 state 0x2 ref 0 00000100:00100000:2.0:1789760774.097237:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760774.097792:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484320768 00000100:00100000:1.0:1789760774.097805:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484320768 00000100:00100000:1.0:1789760774.097973:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763332015022 time 763386759709 state 0x0 ref 1 00000100:00100000:1.0:1789760774.097996:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763387020125 time 763386759709 state 0x1 ref 1 00000100:00100000:1.0:1789760774.098099:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763387020125 time 763429028826 state 0x0 ref 1 00000100:00100000:1.0:1789760774.098114:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484320768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760774.098175:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24047616, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760774.098185:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.098205:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760774.098210:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760774.098217:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24047616->24051711] (req 24047616->24051711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.098233:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24047616->24051711] (req 24047616->24051711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760774.134414:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973189 is committed 00002000:00010000:3.0:1789760774.134584:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24047616 -> 24051712 00010000:00010000:3.0:1789760774.134604:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24047616->24051711] (req 24047616->24051711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.134624:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24047616->24051711] (req 24047616->24051711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.134634:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24047616->24051711] (req 24047616->24051711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.134641:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760774.134646:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24047616->24051711] (req 24047616->24051711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.134657:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24047616->24051711] (req 24047616->24051711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.134682:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24047616->24051711] (req 24047616->24051711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.134689:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209a46 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24047616->24051711] (req 24047616->24051711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760774.134817:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484320768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36698us (37588us total) trans 4294973189 rc 0/0 00000100:00100000:3.0:1789760774.134829:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5900 00000100:00100000:3.0:1789760774.134840:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763387020125 time 763429028826 state 0x2 ref 0 00000100:00100000:0.0:1789760774.137872:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760774.138015:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484320896 00000100:00100000:1.0:1789760774.138028:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484320896 00000100:00100000:1.0:1789760774.138142:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763387020125 time 763429028826 state 0x0 ref 1 00000100:00100000:1.0:1789760774.138168:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763429289242 time 763429028826 state 0x1 ref 1 00000100:00100000:1.0:1789760774.138197:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763429289242 time 763469128166 state 0x0 ref 1 00000100:00100000:1.0:1789760774.138210:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484320896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760774.138270:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24051712, mtime: 0x6aad9505, atime: 0x0, ctime: 0x6aad9505, blocks: 0xb807 00010000:00010000:1.0:1789760774.138280:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.138299:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760774.138304:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760774.138311:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24051712->24055807] (req 24051712->24055807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.138327:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24051712->24055807] (req 24051712->24055807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760774.181136:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973190 is committed 00002000:00010000:3.0:1789760774.181448:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24051712 -> 24055808 00002000:00010000:3.0:1789760774.181455:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760773 -> 1789760774 00002000:00010000:3.0:1789760774.181458:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760773 -> 1789760774 00010000:00010000:3.0:1789760774.181474:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24051712->24055807] (req 24051712->24055807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.181490:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24051712->24055807] (req 24051712->24055807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.181560:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24051712->24055807] (req 24051712->24055807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.181569:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760774.181575:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24051712->24055807] (req 24051712->24055807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.181586:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24051712->24055807] (req 24051712->24055807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.181612:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24051712->24055807] (req 24051712->24055807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.181620:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209a4d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24051712->24055807] (req 24051712->24055807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760774.181874:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484320896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43658us (44016us total) trans 4294973190 rc 0/0 00000100:00100000:3.0:1789760774.181886:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5901 00000100:00100000:3.0:1789760774.181899:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763429289242 time 763469128166 state 0x2 ref 0 00000100:00100000:0.0:1789760774.184644:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760774.184733:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484321024 00000100:00100000:3.0:1789760774.184745:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484321024 00000100:00100000:3.0:1789760774.184813:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763429289242 time 763469128166 state 0x0 ref 1 00000100:00100000:3.0:1789760774.184832:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763469388582 time 763469128166 state 0x1 ref 1 00000100:00100000:3.0:1789760774.184912:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763469388582 time 763515841722 state 0x0 ref 1 00000100:00100000:3.0:1789760774.184926:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484321024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760774.184990:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24055808, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:3.0:1789760774.185000:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.185025:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:3.0:1789760774.185029:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760774.185037:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24055808->24059903] (req 24055808->24059903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.185053:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24055808->24059903] (req 24055808->24059903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760774.236132:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973191 is committed 00002000:00010000:1.0:1789760774.236624:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24055808 -> 24059904 00010000:00010000:1.0:1789760774.236645:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24055808->24059903] (req 24055808->24059903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.236660:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24055808->24059903] (req 24055808->24059903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.236669:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24055808->24059903] (req 24055808->24059903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.236676:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760774.236680:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24055808->24059903] (req 24055808->24059903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.236695:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24055808->24059903] (req 24055808->24059903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.236721:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24055808->24059903] (req 24055808->24059903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.236727:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209a54 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24055808->24059903] (req 24055808->24059903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760774.236945:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484321024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52014us (52305us total) trans 4294973191 rc 0/0 00000100:00100000:1.0:1789760774.236957:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5902 00000100:00100000:1.0:1789760774.237042:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763469388582 time 763515841722 state 0x2 ref 0 00000100:00100000:2.0:1789760774.240407:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760774.240565:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484321152 00000100:00100000:0.0:1789760774.240579:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484321152 00000100:00100000:0.0:1789760774.240651:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763469388582 time 763515841722 state 0x0 ref 1 00000100:00100000:0.0:1789760774.240668:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763516102138 time 763515841722 state 0x1 ref 1 00000100:00100000:0.0:1789760774.240699:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763516102138 time 763571630265 state 0x0 ref 1 00000100:00100000:0.0:1789760774.240713:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484321152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760774.240771:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24059904, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:0.0:1789760774.240782:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.240801:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:0.0:1789760774.240806:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760774.240814:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24059904->24063999] (req 24059904->24063999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.240830:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24059904->24063999] (req 24059904->24063999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760774.305587:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973192 is committed 00002000:00010000:0.0:1789760774.305884:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24059904 -> 24064000 00010000:00010000:0.0:1789760774.305906:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24059904->24063999] (req 24059904->24063999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.305930:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24059904->24063999] (req 24059904->24063999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.305941:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24059904->24063999] (req 24059904->24063999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.305949:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760774.305955:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24059904->24063999] (req 24059904->24063999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.305966:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24059904->24063999] (req 24059904->24063999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.306004:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24059904->24063999] (req 24059904->24063999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.306012:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209a5b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24059904->24063999] (req 24059904->24063999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760774.306203:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484321152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65485us (65800us total) trans 4294973192 rc 0/0 00000100:00100000:0.0:1789760774.306216:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5903 00000100:00100000:0.0:1789760774.306228:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763516102138 time 763571630265 state 0x2 ref 0 00000100:00100000:0.0:1789760774.314001:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760774.314495:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484321280 00000100:00100000:1.0:1789760774.314509:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484321280 00000100:00100000:1.0:1789760774.314590:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763516102138 time 763571630265 state 0x0 ref 1 00000100:00100000:1.0:1789760774.314609:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763571890681 time 763571630265 state 0x1 ref 1 00000100:00100000:1.0:1789760774.314648:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763571890681 time 763645577999 state 0x0 ref 1 00000100:00100000:1.0:1789760774.314662:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484321280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760774.314738:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24064000, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:1.0:1789760774.314750:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.314770:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760774.314775:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760774.314782:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24064000->24068095] (req 24064000->24068095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.314799:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24064000->24068095] (req 24064000->24068095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760774.348658:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24064000 -> 24068096 00010000:00010000:1.0:1789760774.348689:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24064000->24068095] (req 24064000->24068095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.348705:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24064000->24068095] (req 24064000->24068095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.348715:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24064000->24068095] (req 24064000->24068095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.348724:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760774.348730:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24064000->24068095] (req 24064000->24068095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.348741:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24064000->24068095] (req 24064000->24068095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.348770:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24064000->24068095] (req 24064000->24068095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.348779:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a62 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24064000->24068095] (req 24064000->24068095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760774.349208:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484321280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34537us (35213us total) trans 4294973193 rc 0/0 00000100:00100000:1.0:1789760774.349221:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5904 00000100:00100000:1.0:1789760774.349233:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763571890681 time 763645577999 state 0x2 ref 0 00000001:00080000:1.0:1789760774.349290:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973193 is committed 00000100:00100000:2.0:1789760774.353863:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760774.353927:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484321408 00000100:00100000:3.0:1789760774.353941:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484321408 00000100:00100000:3.0:1789760774.354012:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763571890681 time 763645577999 state 0x0 ref 1 00000100:00100000:3.0:1789760774.354033:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763645838415 time 763645577999 state 0x1 ref 1 00000100:00100000:3.0:1789760774.354117:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763645838415 time 763685046573 state 0x0 ref 1 00000100:00100000:3.0:1789760774.354145:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484321408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760774.354684:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24068096, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:3.0:1789760774.354697:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.354718:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:3.0:1789760774.354723:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760774.354732:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24068096->24072191] (req 24068096->24072191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.354750:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24068096->24072191] (req 24068096->24072191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760774.396832:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973194 is committed 00002000:00010000:0.0:1789760774.397166:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24068096 -> 24072192 00010000:00010000:0.0:1789760774.397192:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24068096->24072191] (req 24068096->24072191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.397208:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24068096->24072191] (req 24068096->24072191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.397218:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24068096->24072191] (req 24068096->24072191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.397226:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760774.397231:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24068096->24072191] (req 24068096->24072191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.397242:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24068096->24072191] (req 24068096->24072191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.397270:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24068096->24072191] (req 24068096->24072191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.397277:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209a69 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24068096->24072191] (req 24068096->24072191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760774.397410:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484321408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43262us (43555us total) trans 4294973194 rc 0/0 00000100:00100000:0.0:1789760774.397424:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5905 00000100:00100000:0.0:1789760774.397435:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763645838415 time 763685046573 state 0x2 ref 0 00000100:00100000:3.0:1789760774.402939:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760774.403841:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484321536 00000100:00100000:1.0:1789760774.403852:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484321536 00000100:00100000:1.0:1789760774.403908:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763645838415 time 763685046573 state 0x0 ref 1 00000100:00100000:1.0:1789760774.403922:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763685306989 time 763685046573 state 0x1 ref 1 00000100:00100000:1.0:1789760774.403956:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763685306989 time 763734887566 state 0x0 ref 1 00000100:00100000:1.0:1789760774.403969:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484321536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760774.404017:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24072192, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:1.0:1789760774.404025:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.404046:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760774.404051:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760774.404056:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24072192->24076287] (req 24072192->24076287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.404180:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24072192->24076287] (req 24072192->24076287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760774.442705:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973195 is committed 00002000:00010000:2.0:1789760774.442803:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24072192 -> 24076288 00010000:00010000:2.0:1789760774.442818:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24072192->24076287] (req 24072192->24076287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.442830:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24072192->24076287] (req 24072192->24076287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.442839:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24072192->24076287] (req 24072192->24076287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.442846:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760774.442862:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24072192->24076287] (req 24072192->24076287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.442872:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24072192->24076287] (req 24072192->24076287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.442892:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24072192->24076287] (req 24072192->24076287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.442898:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209a70 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24072192->24076287] (req 24072192->24076287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760774.442988:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484321536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39016us (40054us total) trans 4294973195 rc 0/0 00000100:00100000:2.0:1789760774.442998:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5906 00000100:00100000:2.0:1789760774.443007:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763685306989 time 763734887566 state 0x2 ref 0 00000100:00100000:2.0:1789760774.445822:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760774.445971:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484321664 00000100:00100000:0.0:1789760774.445982:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484321664 00000100:00100000:0.0:1789760774.446040:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763685306989 time 763734887566 state 0x0 ref 1 00000100:00100000:0.0:1789760774.446055:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763735147982 time 763734887566 state 0x1 ref 1 00000100:00100000:0.0:1789760774.446200:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763735147982 time 763777130814 state 0x0 ref 1 00000100:00100000:0.0:1789760774.446215:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484321664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760774.446263:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24076288, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:0.0:1789760774.446271:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.446287:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:0.0:1789760774.446291:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760774.446298:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24076288->24080383] (req 24076288->24080383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.446311:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24076288->24080383] (req 24076288->24080383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760774.481164:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973196 is committed 00002000:00010000:0.0:1789760774.481372:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24076288 -> 24080384 00010000:00010000:0.0:1789760774.481388:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24076288->24080383] (req 24076288->24080383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.481405:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24076288->24080383] (req 24076288->24080383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.481412:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24076288->24080383] (req 24076288->24080383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.481420:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760774.481424:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24076288->24080383] (req 24076288->24080383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.481433:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24076288->24080383] (req 24076288->24080383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.481451:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24076288->24080383] (req 24076288->24080383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.481537:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209a77 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24076288->24080383] (req 24076288->24080383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760774.481679:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484321664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35462us (35860us total) trans 4294973196 rc 0/0 00000100:00100000:0.0:1789760774.481688:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5907 00000100:00100000:0.0:1789760774.481697:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763735147982 time 763777130814 state 0x2 ref 0 00000100:00100000:0.0:1789760774.486874:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760774.487268:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484321792 00000100:00100000:3.0:1789760774.487278:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484321792 00000100:00100000:3.0:1789760774.487331:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763735147982 time 763777130814 state 0x0 ref 1 00000100:00100000:3.0:1789760774.487346:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763777391230 time 763777130814 state 0x1 ref 1 00000100:00100000:3.0:1789760774.487374:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763777391230 time 763818305383 state 0x0 ref 1 00000100:00100000:3.0:1789760774.487387:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484321792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760774.487435:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24080384, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:3.0:1789760774.487444:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.487626:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:3.0:1789760774.487632:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760774.487639:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24080384->24084479] (req 24080384->24084479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.487655:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24080384->24084479] (req 24080384->24084479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760774.519907:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24080384 -> 24084480 00010000:00010000:3.0:1789760774.519927:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24080384->24084479] (req 24080384->24084479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.519939:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24080384->24084479] (req 24080384->24084479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.519946:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24080384->24084479] (req 24080384->24084479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.519952:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760774.519957:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24080384->24084479] (req 24080384->24084479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.519965:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24080384->24084479] (req 24080384->24084479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.519984:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24080384->24084479] (req 24080384->24084479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.519991:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209a7e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24080384->24084479] (req 24080384->24084479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760774.520135:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484321792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32737us (33256us total) trans 4294973197 rc 0/0 00000100:00100000:3.0:1789760774.520146:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5908 00000100:00100000:3.0:1789760774.520334:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763777391230 time 763818305383 state 0x2 ref 0 00000001:00080000:0.0:1789760774.521461:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973197 is committed 00000100:00100000:0.0:1789760774.525648:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760774.525753:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484321920 00000100:00100000:3.0:1789760774.525763:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484321920 00000100:00100000:3.0:1789760774.525820:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763777391230 time 763818305383 state 0x0 ref 1 00000100:00100000:3.0:1789760774.525834:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763818565799 time 763818305383 state 0x1 ref 1 00000100:00100000:3.0:1789760774.525900:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763818565799 time 763856829855 state 0x0 ref 1 00000100:00100000:3.0:1789760774.525913:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484321920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760774.525959:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24084480, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:3.0:1789760774.525968:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.525983:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:3.0:1789760774.525988:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760774.525996:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24084480->24088575] (req 24084480->24088575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.526009:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24084480->24088575] (req 24084480->24088575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760774.541213:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.541235:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9653b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508936448:0@lo:41:osp-pre-1-0.0 00000100:00100000:0.0:1789760774.541306:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760774.541453:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508936448 00000100:00100000:1.0:1789760774.541461:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508936448 00000100:00100000:1.0:1789760774.541510:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 466 00000100:00100000:1.0:1789760774.541519:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508936448:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760774.541553:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760774.541683:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508936448:12345-0@lo:41:osp-pre-1-0.0 Request processed in 164us (377us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760774.541691:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 466 00000100:00100000:3.0:1789760774.541945:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9653b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508936448:0@lo:41:osp-pre-1-0.0 00002000:00010000:0.0:1789760774.560594:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24084480 -> 24088576 00010000:00010000:0.0:1789760774.560624:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24084480->24088575] (req 24084480->24088575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.560637:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24084480->24088575] (req 24084480->24088575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.560646:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24084480->24088575] (req 24084480->24088575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.560653:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760774.560658:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24084480->24088575] (req 24084480->24088575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.560669:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24084480->24088575] (req 24084480->24088575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.560691:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24084480->24088575] (req 24084480->24088575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.560700:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209a85 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24084480->24088575] (req 24084480->24088575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760774.560918:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484321920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35004us (35276us total) trans 4294973198 rc 0/0 00000100:00100000:0.0:1789760774.560927:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5909 00000100:00100000:0.0:1789760774.560934:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763818565799 time 763856829855 state 0x2 ref 0 00000001:00080000:0.0:1789760774.560971:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973198 is committed 00000100:00100000:2.0:1789760774.564351:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760774.564452:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484322048 00000100:00100000:1.0:1789760774.564461:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484322048 00000100:00100000:1.0:1789760774.564509:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763818565799 time 763856829855 state 0x0 ref 1 00000100:00100000:1.0:1789760774.564522:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763857090271 time 763856829855 state 0x1 ref 1 00000100:00100000:1.0:1789760774.564544:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763857090271 time 763895475280 state 0x0 ref 1 00000100:00100000:1.0:1789760774.564554:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484322048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760774.564599:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24088576, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:1.0:1789760774.564608:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.564624:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760774.564628:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760774.564633:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24088576->24092671] (req 24088576->24092671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.564648:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24088576->24092671] (req 24088576->24092671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760774.598167:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973199 is committed 00002000:00010000:0.0:1789760774.598490:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24088576 -> 24092672 00010000:00010000:0.0:1789760774.598505:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24088576->24092671] (req 24088576->24092671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.598583:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24088576->24092671] (req 24088576->24092671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.598592:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24088576->24092671] (req 24088576->24092671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.598600:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760774.598604:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24088576->24092671] (req 24088576->24092671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.598614:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24088576->24092671] (req 24088576->24092671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.598634:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24088576->24092671] (req 24088576->24092671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.598647:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209a8c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24088576->24092671] (req 24088576->24092671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760774.598836:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484322048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34278us (34488us total) trans 4294973199 rc 0/0 00000100:00100000:0.0:1789760774.598846:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5910 00000100:00100000:0.0:1789760774.598867:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763857090271 time 763895475280 state 0x2 ref 0 00000100:00100000:2.0:1789760774.602606:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760774.603459:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484322176 00000100:00100000:0.0:1789760774.603470:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484322176 00000100:00100000:2.0:1789760774.603799:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763857090271 time 763895475280 state 0x0 ref 1 00000100:00100000:2.0:1789760774.603813:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763895735696 time 763895475280 state 0x1 ref 1 00000100:00100000:2.0:1789760774.603961:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763895735696 time 763934891277 state 0x0 ref 1 00000100:00100000:2.0:1789760774.603978:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484322176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760774.604024:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24092672, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:2.0:1789760774.604032:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.604046:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:2.0:1789760774.604050:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760774.604055:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24092672->24096767] (req 24092672->24096767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.604097:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24092672->24096767] (req 24092672->24096767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760774.639298:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973200 is committed 00002000:00010000:0.0:1789760774.639449:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24092672 -> 24096768 00010000:00010000:0.0:1789760774.639540:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24092672->24096767] (req 24092672->24096767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.639555:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24092672->24096767] (req 24092672->24096767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.639565:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24092672->24096767] (req 24092672->24096767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.639572:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760774.639576:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24092672->24096767] (req 24092672->24096767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.639585:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24092672->24096767] (req 24092672->24096767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.639611:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24092672->24096767] (req 24092672->24096767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.639619:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209a93 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24092672->24096767] (req 24092672->24096767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760774.639754:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484322176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35773us (37151us total) trans 4294973200 rc 0/0 00000100:00100000:0.0:1789760774.639764:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5911 00000100:00100000:0.0:1789760774.639774:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763895735696 time 763934891277 state 0x2 ref 0 00000100:00100000:3.0:1789760774.644589:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760774.645576:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484322304 00000100:00100000:1.0:1789760774.645587:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484322304 00000100:00100000:1.0:1789760774.645643:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763895735696 time 763934891277 state 0x0 ref 1 00000100:00100000:1.0:1789760774.645658:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763935151693 time 763934891277 state 0x1 ref 1 00000100:00100000:1.0:1789760774.645679:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763935151693 time 763976610080 state 0x0 ref 1 00000100:00100000:1.0:1789760774.645691:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484322304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760774.645739:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24096768, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:1.0:1789760774.645747:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.645761:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=763s v=(0 0) 00010000:00010000:1.0:1789760774.645766:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760774.645772:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24096768->24100863] (req 24096768->24100863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.645784:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24096768->24100863] (req 24096768->24100863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760774.682732:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973201 is committed 00002000:00010000:3.0:1789760774.682848:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24096768 -> 24100864 00010000:00010000:3.0:1789760774.682861:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24096768->24100863] (req 24096768->24100863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.682934:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24096768->24100863] (req 24096768->24100863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.682944:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24096768->24100863] (req 24096768->24100863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.682951:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760774.682956:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24096768->24100863] (req 24096768->24100863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.682965:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24096768->24100863] (req 24096768->24100863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.683056:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24096768->24100863] (req 24096768->24100863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.683143:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209a9a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24096768->24100863] (req 24096768->24100863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760774.683283:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484322304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37589us (38699us total) trans 4294973201 rc 0/0 00000100:00100000:3.0:1789760774.683294:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5912 00000100:00100000:3.0:1789760774.683304:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763935151693 time 763976610080 state 0x2 ref 0 00000100:00100000:3.0:1789760774.692704:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760774.692805:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484322432 00000100:00100000:0.0:1789760774.692816:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484322432 00000100:00100000:0.0:1789760774.692938:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763935151693 time 763976610080 state 0x0 ref 1 00000100:00100000:0.0:1789760774.692954:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763976870496 time 763976610080 state 0x1 ref 1 00000100:00100000:0.0:1789760774.693113:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763976870496 time 764024043014 state 0x0 ref 1 00000100:00100000:0.0:1789760774.693128:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484322432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760774.693177:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24100864, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:0.0:1789760774.693186:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.693207:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:0.0:1789760774.693212:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760774.693218:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24100864->24104959] (req 24100864->24104959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.693231:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24100864->24104959] (req 24100864->24104959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760774.733268:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973202 is committed 00002000:00010000:1.0:1789760774.733543:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24100864 -> 24104960 00010000:00010000:1.0:1789760774.733558:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24100864->24104959] (req 24100864->24104959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.733572:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24100864->24104959] (req 24100864->24104959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.733582:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24100864->24104959] (req 24100864->24104959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.733589:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760774.733594:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24100864->24104959] (req 24100864->24104959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.733604:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24100864->24104959] (req 24100864->24104959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.733625:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24100864->24104959] (req 24100864->24104959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.733633:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209aa1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24100864->24104959] (req 24100864->24104959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760774.733731:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484322432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40602us (41034us total) trans 4294973202 rc 0/0 00000100:00100000:1.0:1789760774.733742:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5913 00000100:00100000:1.0:1789760774.733752:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763976870496 time 764024043014 state 0x2 ref 0 00000100:00100000:2.0:1789760774.740479:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760774.740917:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484322560 00000100:00100000:0.0:1789760774.740927:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484322560 00000100:00100000:0.0:1789760774.741058:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 763976870496 time 764024043014 state 0x0 ref 1 00000100:00100000:0.0:1789760774.741175:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764024303430 time 764024043014 state 0x1 ref 1 00000100:00100000:0.0:1789760774.741258:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764024303430 time 764072188079 state 0x0 ref 1 00000100:00100000:0.0:1789760774.741275:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484322560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760774.741325:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24104960, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:0.0:1789760774.741334:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.741368:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:0.0:1789760774.741373:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760774.741380:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24104960->24109055] (req 24104960->24109055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.741400:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24104960->24109055] (req 24104960->24109055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760774.781660:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24104960 -> 24109056 00010000:00010000:2.0:1789760774.781679:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24104960->24109055] (req 24104960->24109055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.781693:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24104960->24109055] (req 24104960->24109055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.781704:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24104960->24109055] (req 24104960->24109055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.781711:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760774.781716:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24104960->24109055] (req 24104960->24109055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.781726:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24104960->24109055] (req 24104960->24109055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.781746:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24104960->24109055] (req 24104960->24109055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.781755:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209aa8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24104960->24109055] (req 24104960->24109055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760774.781864:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484322560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40580us (41381us total) trans 4294973203 rc 0/0 00000100:00100000:2.0:1789760774.781875:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5914 00000100:00100000:2.0:1789760774.781885:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764024303430 time 764072188079 state 0x2 ref 0 00000001:00080000:3.0:1789760774.782098:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973203 is committed 00000100:00100000:3.0:1789760774.786209:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760774.786271:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484322688 00000100:00100000:2.0:1789760774.786281:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484322688 00000100:00100000:2.0:1789760774.786336:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764024303430 time 764072188079 state 0x0 ref 1 00000100:00100000:2.0:1789760774.786351:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764072448495 time 764072188079 state 0x1 ref 1 00000100:00100000:2.0:1789760774.786448:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764072448495 time 764117377748 state 0x0 ref 1 00000100:00100000:2.0:1789760774.786462:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484322688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760774.786509:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24109056, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:2.0:1789760774.786518:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.786533:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:2.0:1789760774.786538:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760774.786544:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24109056->24113151] (req 24109056->24113151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.786557:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24109056->24113151] (req 24109056->24113151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760774.837108:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973204 is committed 00002000:00010000:2.0:1789760774.837531:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24109056 -> 24113152 00010000:00010000:2.0:1789760774.837546:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24109056->24113151] (req 24109056->24113151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.837560:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24109056->24113151] (req 24109056->24113151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.837569:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24109056->24113151] (req 24109056->24113151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.837577:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760774.837582:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24109056->24113151] (req 24109056->24113151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.837591:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24109056->24113151] (req 24109056->24113151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.837611:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24109056->24113151] (req 24109056->24113151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.837619:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209aaf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24109056->24113151] (req 24109056->24113151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760774.837798:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484322688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51332us (51592us total) trans 4294973204 rc 0/0 00000100:00100000:2.0:1789760774.837808:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5915 00000100:00100000:2.0:1789760774.837817:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764072448495 time 764117377748 state 0x2 ref 0 00000100:00100000:0.0:1789760774.841446:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760774.841770:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484322816 00000100:00100000:3.0:1789760774.841780:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484322816 00000100:00100000:3.0:1789760774.841840:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764072448495 time 764117377748 state 0x0 ref 1 00000100:00100000:3.0:1789760774.841932:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764117638164 time 764117377748 state 0x1 ref 1 00000100:00100000:3.0:1789760774.842000:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764117638164 time 764172930723 state 0x0 ref 1 00000100:00100000:3.0:1789760774.842015:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484322816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760774.842062:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24113152, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:3.0:1789760774.842102:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.842117:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:3.0:1789760774.842122:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760774.842128:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24113152->24117247] (req 24113152->24117247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.842143:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24113152->24117247] (req 24113152->24117247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760774.883053:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973205 is committed 00002000:00010000:2.0:1789760774.883236:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24113152 -> 24117248 00010000:00010000:2.0:1789760774.883251:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24113152->24117247] (req 24113152->24117247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.883264:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24113152->24117247] (req 24113152->24117247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.883273:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24113152->24117247] (req 24113152->24117247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.883281:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760774.883286:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24113152->24117247] (req 24113152->24117247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.883297:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24113152->24117247] (req 24113152->24117247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.883321:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24113152->24117247] (req 24113152->24117247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.883328:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209ab6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24113152->24117247] (req 24113152->24117247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760774.883426:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484322816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41410us (41985us total) trans 4294973205 rc 0/0 00000100:00100000:2.0:1789760774.883438:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5916 00000100:00100000:2.0:1789760774.883448:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764117638164 time 764172930723 state 0x2 ref 0 00000100:00100000:0.0:1789760774.889328:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760774.889479:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484322944 00000100:00100000:1.0:1789760774.889489:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484322944 00000100:00100000:1.0:1789760774.889545:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764117638164 time 764172930723 state 0x0 ref 1 00000100:00100000:1.0:1789760774.889561:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764173191139 time 764172930723 state 0x1 ref 1 00000100:00100000:1.0:1789760774.889632:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764173191139 time 764220561661 state 0x0 ref 1 00000100:00100000:1.0:1789760774.889644:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484322944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760774.889692:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24117248, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xb807 00010000:00010000:1.0:1789760774.889700:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.889717:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:1.0:1789760774.889720:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760774.889726:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24117248->24121343] (req 24117248->24121343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760774.889744:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24117248->24121343] (req 24117248->24121343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760774.922484:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973206 is committed 00002000:00010000:3.0:1789760774.922709:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24117248 -> 24121344 00002000:00010000:3.0:1789760774.922714:0:8108:0:(ofd_lvb.c:331:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb blocks from disk: 47111 -> 49159 00010000:00010000:3.0:1789760774.922726:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24117248->24121343] (req 24117248->24121343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.922740:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24117248->24121343] (req 24117248->24121343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.922749:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24117248->24121343] (req 24117248->24121343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.922756:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760774.922760:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24117248->24121343] (req 24117248->24121343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.922768:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24117248->24121343] (req 24117248->24121343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.922788:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24117248->24121343] (req 24117248->24121343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760774.922795:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209abd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24117248->24121343] (req 24117248->24121343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760774.922950:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484322944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33303us (33628us total) trans 4294973206 rc 0/0 00000100:00100000:3.0:1789760774.922960:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5917 00000100:00100000:3.0:1789760774.922969:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764173191139 time 764220561661 state 0x2 ref 0 00000100:00100000:2.0:1789760774.927001:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760774.927249:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484323072 00000100:00100000:0.0:1789760774.927259:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484323072 00000100:00100000:0.0:1789760774.927321:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764173191139 time 764220561661 state 0x0 ref 1 00000100:00100000:0.0:1789760774.927336:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764220822077 time 764220561661 state 0x1 ref 1 00000100:00100000:0.0:1789760774.927459:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764220822077 time 764258388579 state 0x0 ref 1 00000100:00100000:0.0:1789760774.927473:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484323072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760774.927532:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24121344, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xc007 00010000:00010000:0.0:1789760774.927542:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.927558:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:0.0:1789760774.927561:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760774.927567:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24121344->24125439] (req 24121344->24125439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.927580:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24121344->24125439] (req 24121344->24125439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760774.959792:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24121344 -> 24125440 00010000:00010000:0.0:1789760774.959812:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24121344->24125439] (req 24121344->24125439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.959825:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24121344->24125439] (req 24121344->24125439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.959834:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24121344->24125439] (req 24121344->24125439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.959841:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760774.959846:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24121344->24125439] (req 24121344->24125439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.959855:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24121344->24125439] (req 24121344->24125439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.959875:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24121344->24125439] (req 24121344->24125439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760774.959883:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209ac4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24121344->24125439] (req 24121344->24125439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760774.959971:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484323072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32497us (32974us total) trans 4294973207 rc 0/0 00000100:00100000:0.0:1789760774.959985:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5918 00000100:00100000:0.0:1789760774.959994:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764220822077 time 764258388579 state 0x2 ref 0 00000001:00080000:2.0:1789760774.960099:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973207 is committed 00000100:00100000:0.0:1789760774.964983:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760774.965110:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484323200 00000100:00100000:2.0:1789760774.965121:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484323200 00000100:00100000:2.0:1789760774.965249:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764220822077 time 764258388579 state 0x0 ref 1 00000100:00100000:2.0:1789760774.965264:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764258648995 time 764258388579 state 0x1 ref 1 00000100:00100000:2.0:1789760774.965297:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764258648995 time 764296227960 state 0x0 ref 1 00000100:00100000:2.0:1789760774.965310:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484323200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760774.965357:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24125440, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xc007 00010000:00010000:2.0:1789760774.965365:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.965456:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:2.0:1789760774.965461:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760774.965468:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24125440->24129535] (req 24125440->24129535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760774.965483:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24125440->24129535] (req 24125440->24129535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760775.003525:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973208 is committed 00002000:00010000:0.0:1789760775.003756:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24125440 -> 24129536 00010000:00010000:0.0:1789760775.003770:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24125440->24129535] (req 24125440->24129535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.003782:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24125440->24129535] (req 24125440->24129535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.003790:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24125440->24129535] (req 24125440->24129535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.003796:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760775.003801:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24125440->24129535] (req 24125440->24129535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.003809:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24125440->24129535] (req 24125440->24129535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.003829:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24125440->24129535] (req 24125440->24129535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.003836:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209acb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24125440->24129535] (req 24125440->24129535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760775.003945:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484323200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38634us (38968us total) trans 4294973208 rc 0/0 00000100:00100000:0.0:1789760775.003956:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5919 00000100:00100000:0.0:1789760775.003965:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764258648995 time 764296227960 state 0x2 ref 0 00000100:00100000:1.0:1789760775.008040:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.008277:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484323328 00000100:00100000:2.0:1789760775.008288:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484323328 00000100:00100000:2.0:1789760775.008345:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764258648995 time 764296227960 state 0x0 ref 1 00000100:00100000:2.0:1789760775.008361:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764296488376 time 764296227960 state 0x1 ref 1 00000100:00100000:2.0:1789760775.008423:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764296488376 time 764339352895 state 0x0 ref 1 00000100:00100000:2.0:1789760775.008435:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9651f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484323328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.008505:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24129536, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xc007 00010000:00010000:2.0:1789760775.008514:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.008531:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:2.0:1789760775.008535:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.008542:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24129536->24133631] (req 24129536->24133631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.008561:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24129536->24133631] (req 24129536->24133631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760775.039955:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24129536 -> 24133632 00010000:00010000:1.0:1789760775.039976:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24129536->24133631] (req 24129536->24133631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.039990:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24129536->24133631] (req 24129536->24133631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.040000:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24129536->24133631] (req 24129536->24133631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.040007:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760775.040012:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24129536->24133631] (req 24129536->24133631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.040021:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24129536->24133631] (req 24129536->24133631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.040214:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24129536->24133631] (req 24129536->24133631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.040231:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ad2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24129536->24133631] (req 24129536->24133631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760775.040316:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9651f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484323328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31878us (32281us total) trans 4294973209 rc 0/0 00000100:00100000:1.0:1789760775.040326:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5920 00000100:00100000:1.0:1789760775.040337:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764296488376 time 764339352895 state 0x2 ref 0 00000001:00080000:1.0:1789760775.040382:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973209 is committed 00000100:00100000:2.0:1789760775.045511:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760775.045577:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484323456 00000100:00100000:3.0:1789760775.045587:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484323456 00000100:00100000:3.0:1789760775.045639:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764296488376 time 764339352895 state 0x0 ref 1 00000100:00100000:3.0:1789760775.045655:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764339613311 time 764339352895 state 0x1 ref 1 00000100:00100000:3.0:1789760775.045684:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764339613311 time 764376615441 state 0x0 ref 1 00000100:00100000:3.0:1789760775.045697:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484323456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760775.045745:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24133632, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xc007 00010000:00010000:3.0:1789760775.045755:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.045771:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:3.0:1789760775.045776:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760775.045783:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24133632->24137727] (req 24133632->24137727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.045797:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24133632->24137727] (req 24133632->24137727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.055330:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760775.089506:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973210 is committed 00002000:00010000:1.0:1789760775.089627:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24133632 -> 24137728 00010000:00010000:1.0:1789760775.089643:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24133632->24137727] (req 24133632->24137727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.089656:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24133632->24137727] (req 24133632->24137727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.089665:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24133632->24137727] (req 24133632->24137727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.089672:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760775.089678:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24133632->24137727] (req 24133632->24137727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.089688:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24133632->24137727] (req 24133632->24137727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.089714:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24133632->24137727] (req 24133632->24137727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.089721:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209ad9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24133632->24137727] (req 24133632->24137727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760775.089858:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484323456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44150us (44353us total) trans 4294973210 rc 0/0 00000100:00100000:1.0:1789760775.089869:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5921 00000100:00100000:1.0:1789760775.089878:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764339613311 time 764376615441 state 0x2 ref 0 00000100:00100000:0.0:1789760775.098241:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.098475:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484323584 00000100:00100000:2.0:1789760775.098484:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484323584 00000100:00100000:2.0:1789760775.098535:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764339613311 time 764376615441 state 0x0 ref 1 00000100:00100000:2.0:1789760775.098550:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764376875857 time 764376615441 state 0x1 ref 1 00000100:00100000:2.0:1789760775.098580:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764376875857 time 764429511788 state 0x0 ref 1 00000100:00100000:2.0:1789760775.098594:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484323584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.098641:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24137728, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xc007 00010000:00010000:2.0:1789760775.098650:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.098665:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:2.0:1789760775.098669:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.098676:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24137728->24141823] (req 24137728->24141823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.098691:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24137728->24141823] (req 24137728->24141823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760775.127519:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24137728 -> 24141824 00010000:00010000:3.0:1789760775.127541:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24137728->24141823] (req 24137728->24141823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.127554:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24137728->24141823] (req 24137728->24141823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.127562:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24137728->24141823] (req 24137728->24141823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.127569:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760775.127574:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24137728->24141823] (req 24137728->24141823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.127583:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24137728->24141823] (req 24137728->24141823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.127603:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24137728->24141823] (req 24137728->24141823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.127617:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ae0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24137728->24141823] (req 24137728->24141823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760775.127708:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8911880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484323584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29112us (29647us total) trans 4294973211 rc 0/0 00000100:00100000:3.0:1789760775.127718:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5922 00000100:00100000:3.0:1789760775.127728:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764376875857 time 764429511788 state 0x2 ref 0 00000001:00080000:1.0:1789760775.130514:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973211 is committed 00000100:00100000:0.0:1789760775.131683:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.131771:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484323712 00000100:00100000:2.0:1789760775.131781:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484323712 00000100:00100000:2.0:1789760775.131847:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764376875857 time 764429511788 state 0x0 ref 1 00000100:00100000:2.0:1789760775.131867:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764429772204 time 764429511788 state 0x1 ref 1 00000100:00100000:2.0:1789760775.131896:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764429772204 time 764462827156 state 0x0 ref 1 00000100:00100000:2.0:1789760775.131910:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484323712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.131954:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24141824, mtime: 0x6aad9506, atime: 0x0, ctime: 0x6aad9506, blocks: 0xc007 00010000:00010000:2.0:1789760775.131963:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.131977:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:2.0:1789760775.131982:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.131988:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24141824->24145919] (req 24141824->24145919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.132002:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24141824->24145919] (req 24141824->24145919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760775.166440:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973212 is committed 00002000:00010000:2.0:1789760775.166534:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24141824 -> 24145920 00002000:00010000:2.0:1789760775.166539:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760774 -> 1789760775 00002000:00010000:2.0:1789760775.166542:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760774 -> 1789760775 00010000:00010000:2.0:1789760775.166556:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24141824->24145919] (req 24141824->24145919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.166569:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24141824->24145919] (req 24141824->24145919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.166578:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24141824->24145919] (req 24141824->24145919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.166585:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760775.166591:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24141824->24145919] (req 24141824->24145919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.166600:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24141824->24145919] (req 24141824->24145919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.166620:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24141824->24145919] (req 24141824->24145919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.166628:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209ae7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24141824->24145919] (req 24141824->24145919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760775.166721:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484323712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34809us (35040us total) trans 4294973212 rc 0/0 00000100:00100000:2.0:1789760775.166732:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5923 00000100:00100000:2.0:1789760775.166742:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764429772204 time 764462827156 state 0x2 ref 0 00000100:00100000:2.0:1789760775.171469:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760775.171718:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484323840 00000100:00100000:1.0:1789760775.171729:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484323840 00000100:00100000:1.0:1789760775.171787:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764429772204 time 764462827156 state 0x0 ref 1 00000100:00100000:1.0:1789760775.171802:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764463087572 time 764462827156 state 0x1 ref 1 00000100:00100000:1.0:1789760775.171834:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764463087572 time 764502765129 state 0x0 ref 1 00000100:00100000:1.0:1789760775.171858:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484323840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760775.171906:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24145920, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:1.0:1789760775.171915:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.171935:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:1.0:1789760775.171939:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760775.171945:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24145920->24150015] (req 24145920->24150015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.171958:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24145920->24150015] (req 24145920->24150015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760775.215618:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973213 is committed 00002000:00010000:2.0:1789760775.215786:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24145920 -> 24150016 00010000:00010000:2.0:1789760775.215802:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24145920->24150015] (req 24145920->24150015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.215815:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24145920->24150015] (req 24145920->24150015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.215823:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24145920->24150015] (req 24145920->24150015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.215829:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760775.215834:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24145920->24150015] (req 24145920->24150015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.215842:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24145920->24150015] (req 24145920->24150015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.215871:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24145920->24150015] (req 24145920->24150015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.215878:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209aee lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24145920->24150015] (req 24145920->24150015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760775.216261:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484323840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44411us (44795us total) trans 4294973213 rc 0/0 00000100:00100000:2.0:1789760775.216272:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5924 00000100:00100000:2.0:1789760775.216281:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764463087572 time 764502765129 state 0x2 ref 0 00000100:00100000:3.0:1789760775.220476:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760775.220620:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484323968 00000100:00100000:1.0:1789760775.220630:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484323968 00000100:00100000:1.0:1789760775.220680:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764463087572 time 764502765129 state 0x0 ref 1 00000100:00100000:1.0:1789760775.220694:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764503025545 time 764502765129 state 0x1 ref 1 00000100:00100000:1.0:1789760775.220751:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764503025545 time 764551681666 state 0x0 ref 1 00000100:00100000:1.0:1789760775.220762:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b18380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484323968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760775.220806:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24150016, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:1.0:1789760775.220815:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.220831:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:1.0:1789760775.220835:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760775.220841:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24150016->24154111] (req 24150016->24154111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.220860:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24150016->24154111] (req 24150016->24154111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760775.279497:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973214 is committed 00002000:00010000:1.0:1789760775.279661:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24150016 -> 24154112 00010000:00010000:1.0:1789760775.279676:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24150016->24154111] (req 24150016->24154111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.279689:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24150016->24154111] (req 24150016->24154111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.279699:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24150016->24154111] (req 24150016->24154111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.279706:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760775.279710:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24150016->24154111] (req 24150016->24154111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.279718:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24150016->24154111] (req 24150016->24154111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.279737:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24150016->24154111] (req 24150016->24154111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.279744:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209af5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24150016->24154111] (req 24150016->24154111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760775.279843:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484323968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59077us (59372us total) trans 4294973214 rc 0/0 00000100:00100000:1.0:1789760775.279863:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5925 00000100:00100000:1.0:1789760775.279874:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764503025545 time 764551681666 state 0x2 ref 0 00000100:00100000:3.0:1789760775.283144:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.283350:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484324096 00000100:00100000:2.0:1789760775.283361:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484324096 00000100:00100000:2.0:1789760775.283417:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764503025545 time 764551681666 state 0x0 ref 1 00000100:00100000:2.0:1789760775.283431:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764551942082 time 764551681666 state 0x1 ref 1 00000100:00100000:2.0:1789760775.283462:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764551942082 time 764614392935 state 0x0 ref 1 00000100:00100000:2.0:1789760775.283475:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.283525:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24154112, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:2.0:1789760775.283535:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.283550:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:2.0:1789760775.283555:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.283561:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24154112->24158207] (req 24154112->24158207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.283574:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24154112->24158207] (req 24154112->24158207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760775.317429:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973215 is committed 00002000:00010000:2.0:1789760775.317714:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24154112 -> 24158208 00010000:00010000:2.0:1789760775.317731:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24154112->24158207] (req 24154112->24158207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.317749:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24154112->24158207] (req 24154112->24158207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.317758:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24154112->24158207] (req 24154112->24158207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.317765:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760775.317768:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24154112->24158207] (req 24154112->24158207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.317777:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24154112->24158207] (req 24154112->24158207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.317797:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24154112->24158207] (req 24154112->24158207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.317805:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209afc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24154112->24158207] (req 24154112->24158207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760775.318007:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34529us (34868us total) trans 4294973215 rc 0/0 00000100:00100000:2.0:1789760775.318017:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5926 00000100:00100000:2.0:1789760775.318027:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764551942082 time 764614392935 state 0x2 ref 0 00000100:00100000:1.0:1789760775.324323:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.324414:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484324224 00000100:00100000:2.0:1789760775.324424:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484324224 00000100:00100000:2.0:1789760775.324476:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764551942082 time 764614392935 state 0x0 ref 1 00000100:00100000:2.0:1789760775.324490:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764614653351 time 764614392935 state 0x1 ref 1 00000100:00100000:2.0:1789760775.324520:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764614653351 time 764655451174 state 0x0 ref 1 00000100:00100000:2.0:1789760775.324534:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9653b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484324224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.324582:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24158208, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:2.0:1789760775.324591:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.324605:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:2.0:1789760775.324610:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.324616:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24158208->24162303] (req 24158208->24162303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.324628:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24158208->24162303] (req 24158208->24162303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760775.363135:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24158208 -> 24162304 00000001:00080000:1.0:1789760775.363138:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973216 is committed 00010000:00010000:3.0:1789760775.363244:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24158208->24162303] (req 24158208->24162303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.363258:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24158208->24162303] (req 24158208->24162303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.363266:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24158208->24162303] (req 24158208->24162303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.363273:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760775.363278:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24158208->24162303] (req 24158208->24162303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.363293:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24158208->24162303] (req 24158208->24162303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.363312:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24158208->24162303] (req 24158208->24162303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.363321:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b03 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24158208->24162303] (req 24158208->24162303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760775.363408:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9653b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484324224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38868us (39086us total) trans 4294973216 rc 0/0 00000100:00100000:3.0:1789760775.363418:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5927 00000100:00100000:3.0:1789760775.363427:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764614653351 time 764655451174 state 0x2 ref 0 00000100:00100000:2.0:1789760775.366759:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760775.366887:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484324352 00000100:00100000:1.0:1789760775.366897:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484324352 00000100:00100000:1.0:1789760775.366950:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764614653351 time 764655451174 state 0x0 ref 1 00000100:00100000:1.0:1789760775.366969:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764655711590 time 764655451174 state 0x1 ref 1 00000100:00100000:1.0:1789760775.366990:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764655711590 time 764697921968 state 0x0 ref 1 00000100:00100000:1.0:1789760775.367003:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484324352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760775.367051:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24162304, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:1.0:1789760775.367059:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.367149:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:1.0:1789760775.367154:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760775.367160:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24162304->24166399] (req 24162304->24166399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.367179:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24162304->24166399] (req 24162304->24166399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760775.411687:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973217 is committed 00002000:00010000:2.0:1789760775.411791:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24162304 -> 24166400 00010000:00010000:2.0:1789760775.411805:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24162304->24166399] (req 24162304->24166399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.411817:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24162304->24166399] (req 24162304->24166399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.411832:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24162304->24166399] (req 24162304->24166399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.411840:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760775.411857:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24162304->24166399] (req 24162304->24166399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.411867:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24162304->24166399] (req 24162304->24166399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.411886:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24162304->24166399] (req 24162304->24166399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.411894:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209b0a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24162304->24166399] (req 24162304->24166399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760775.411986:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484324352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44980us (45231us total) trans 4294973217 rc 0/0 00000100:00100000:2.0:1789760775.411996:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5928 00000100:00100000:2.0:1789760775.412007:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764655711590 time 764697921968 state 0x2 ref 0 00000100:00100000:3.0:1789760775.415197:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760775.415343:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484324480 00000100:00100000:1.0:1789760775.415353:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484324480 00000100:00100000:1.0:1789760775.415420:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764655711590 time 764697921968 state 0x0 ref 1 00000100:00100000:1.0:1789760775.415436:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764698182384 time 764697921968 state 0x1 ref 1 00000100:00100000:1.0:1789760775.415466:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764698182384 time 764746397085 state 0x0 ref 1 00000100:00100000:1.0:1789760775.415478:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760775.415526:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24166400, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:1.0:1789760775.415534:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.415551:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:1.0:1789760775.415555:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760775.415562:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24166400->24170495] (req 24166400->24170495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.415576:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24166400->24170495] (req 24166400->24170495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760775.450285:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973218 is committed 00002000:00010000:2.0:1789760775.450604:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24166400 -> 24170496 00010000:00010000:2.0:1789760775.450619:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24166400->24170495] (req 24166400->24170495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.450638:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24166400->24170495] (req 24166400->24170495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.450648:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24166400->24170495] (req 24166400->24170495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.450655:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760775.450660:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24166400->24170495] (req 24166400->24170495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.450670:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24166400->24170495] (req 24166400->24170495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.450688:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24166400->24170495] (req 24166400->24170495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.450696:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209b11 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24166400->24170495] (req 24166400->24170495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760775.450786:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35305us (35594us total) trans 4294973218 rc 0/0 00000100:00100000:2.0:1789760775.450797:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5929 00000100:00100000:2.0:1789760775.450807:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764698182384 time 764746397085 state 0x2 ref 0 00000100:00100000:1.0:1789760775.456323:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.456374:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484324608 00000100:00100000:2.0:1789760775.456385:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484324608 00000100:00100000:2.0:1789760775.456434:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764698182384 time 764746397085 state 0x0 ref 1 00000100:00100000:2.0:1789760775.456449:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764746657501 time 764746397085 state 0x1 ref 1 00000100:00100000:2.0:1789760775.456481:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764746657501 time 764787412449 state 0x0 ref 1 00000100:00100000:2.0:1789760775.456494:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.456541:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24170496, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:2.0:1789760775.456550:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.456567:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:2.0:1789760775.456571:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.456577:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24170496->24174591] (req 24170496->24174591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.456590:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24170496->24174591] (req 24170496->24174591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760775.486518:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973219 is committed 00002000:00010000:2.0:1789760775.486662:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24170496 -> 24174592 00010000:00010000:2.0:1789760775.486678:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24170496->24174591] (req 24170496->24174591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.486691:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24170496->24174591] (req 24170496->24174591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.486707:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24170496->24174591] (req 24170496->24174591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.486715:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760775.486720:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24170496->24174591] (req 24170496->24174591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.486729:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24170496->24174591] (req 24170496->24174591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.486748:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24170496->24174591] (req 24170496->24174591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.486756:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b18 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24170496->24174591] (req 24170496->24174591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760775.486909:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30349us (30530us total) trans 4294973219 rc 0/0 00000100:00100000:2.0:1789760775.486920:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5930 00000100:00100000:2.0:1789760775.486929:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764746657501 time 764787412449 state 0x2 ref 0 00000100:00100000:0.0:1789760775.491280:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760775.491473:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484324736 00000100:00100000:3.0:1789760775.491483:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484324736 00000100:00100000:3.0:1789760775.491533:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764746657501 time 764787412449 state 0x0 ref 1 00000100:00100000:3.0:1789760775.491548:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764787672865 time 764787412449 state 0x1 ref 1 00000100:00100000:3.0:1789760775.491603:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764787672865 time 764822533286 state 0x0 ref 1 00000100:00100000:3.0:1789760775.491616:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760775.491662:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24174592, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:3.0:1789760775.491670:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.491685:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:3.0:1789760775.491689:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760775.491695:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24174592->24178687] (req 24174592->24178687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.491708:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24174592->24178687] (req 24174592->24178687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760775.527633:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973220 is committed 00002000:00010000:3.0:1789760775.527794:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24174592 -> 24178688 00010000:00010000:3.0:1789760775.527808:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24174592->24178687] (req 24174592->24178687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.527822:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24174592->24178687] (req 24174592->24178687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.527831:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24174592->24178687] (req 24174592->24178687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.527838:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760775.527842:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24174592->24178687] (req 24174592->24178687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.527851:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24174592->24178687] (req 24174592->24178687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.527869:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24174592->24178687] (req 24174592->24178687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.527876:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b1f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24174592->24178687] (req 24174592->24178687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760775.528017:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36399us (36740us total) trans 4294973220 rc 0/0 00000100:00100000:3.0:1789760775.528026:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5931 00000100:00100000:3.0:1789760775.528035:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764787672865 time 764822533286 state 0x2 ref 0 00000100:00100000:3.0:1789760775.532205:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760775.532973:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484324864 00000100:00100000:0.0:1789760775.532983:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484324864 00000100:00100000:0.0:1789760775.533051:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764787672865 time 764822533286 state 0x0 ref 1 00000100:00100000:0.0:1789760775.533253:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764822793702 time 764822533286 state 0x1 ref 1 00000100:00100000:0.0:1789760775.533283:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764822793702 time 764864214024 state 0x0 ref 1 00000100:00100000:0.0:1789760775.533297:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484324864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760775.533347:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24178688, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:0.0:1789760775.533356:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.533373:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:0.0:1789760775.533377:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760775.533384:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24178688->24182783] (req 24178688->24182783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.533398:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24178688->24182783] (req 24178688->24182783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760775.566719:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24178688 -> 24182784 00000001:00080000:3.0:1789760775.566731:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973221 is committed 00010000:00010000:1.0:1789760775.566803:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24178688->24182783] (req 24178688->24182783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.566815:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24178688->24182783] (req 24178688->24182783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.566824:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24178688->24182783] (req 24178688->24182783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.566831:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760775.566836:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24178688->24182783] (req 24178688->24182783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.566846:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24178688->24182783] (req 24178688->24182783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.566866:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24178688->24182783] (req 24178688->24182783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760775.566873:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209b26 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24178688->24182783] (req 24178688->24182783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760775.567006:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484324864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33709us (34806us total) trans 4294973221 rc 0/0 00000100:00100000:1.0:1789760775.567017:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5932 00000100:00100000:1.0:1789760775.567027:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764822793702 time 764864214024 state 0x2 ref 0 00000100:00100000:1.0:1789760775.572145:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.572318:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484324992 00000100:00100000:2.0:1789760775.572328:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484324992 00000100:00100000:2.0:1789760775.572380:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764822793702 time 764864214024 state 0x0 ref 1 00000100:00100000:2.0:1789760775.572395:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764864474440 time 764864214024 state 0x1 ref 1 00000100:00100000:2.0:1789760775.572428:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764864474440 time 764903358736 state 0x0 ref 1 00000100:00100000:2.0:1789760775.572440:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.572487:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24182784, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:2.0:1789760775.572495:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.572511:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:2.0:1789760775.572514:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.572520:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24182784->24186879] (req 24182784->24186879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.572533:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24182784->24186879] (req 24182784->24186879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760775.615183:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973222 is committed 00002000:00010000:2.0:1789760775.615360:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24182784 -> 24186880 00010000:00010000:2.0:1789760775.615377:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24182784->24186879] (req 24182784->24186879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.615389:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24182784->24186879] (req 24182784->24186879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.615398:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24182784->24186879] (req 24182784->24186879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.615405:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760775.615411:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24182784->24186879] (req 24182784->24186879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.615420:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24182784->24186879] (req 24182784->24186879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.615437:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24182784->24186879] (req 24182784->24186879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.615444:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b2d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24182784->24186879] (req 24182784->24186879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760775.615535:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484324992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43091us (43396us total) trans 4294973222 rc 0/0 00000100:00100000:2.0:1789760775.615545:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5933 00000100:00100000:2.0:1789760775.615554:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764864474440 time 764903358736 state 0x2 ref 0 00000100:00100000:1.0:1789760775.620690:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760775.620780:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484325120 00000100:00100000:3.0:1789760775.620789:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484325120 00000100:00100000:3.0:1789760775.620856:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764864474440 time 764903358736 state 0x0 ref 1 00000100:00100000:3.0:1789760775.620872:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764903619152 time 764903358736 state 0x1 ref 1 00000100:00100000:3.0:1789760775.620899:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764903619152 time 764951829916 state 0x0 ref 1 00000100:00100000:3.0:1789760775.620910:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484325120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760775.620958:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24186880, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:3.0:1789760775.620967:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.620987:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:3.0:1789760775.620991:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760775.620998:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24186880->24190975] (req 24186880->24190975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.621012:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24186880->24190975] (req 24186880->24190975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760775.647731:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973223 is committed 00002000:00010000:3.0:1789760775.648025:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24186880 -> 24190976 00010000:00010000:3.0:1789760775.648041:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24186880->24190975] (req 24186880->24190975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.648054:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24186880->24190975] (req 24186880->24190975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.648099:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24186880->24190975] (req 24186880->24190975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.648107:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760775.648111:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24186880->24190975] (req 24186880->24190975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.648121:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24186880->24190975] (req 24186880->24190975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.648141:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24186880->24190975] (req 24186880->24190975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.648149:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b34 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24186880->24190975] (req 24186880->24190975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760775.648276:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484325120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27361us (27588us total) trans 4294973223 rc 0/0 00000100:00100000:3.0:1789760775.648287:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5934 00000100:00100000:3.0:1789760775.648298:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764903619152 time 764951829916 state 0x2 ref 0 00000100:00100000:0.0:1789760775.655419:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760775.655474:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484325248 00000100:00100000:3.0:1789760775.655485:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484325248 00000100:00100000:3.0:1789760775.655542:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764903619152 time 764951829916 state 0x0 ref 1 00000100:00100000:3.0:1789760775.655558:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764952090332 time 764951829916 state 0x1 ref 1 00000100:00100000:3.0:1789760775.655775:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764952090332 time 764986705260 state 0x0 ref 1 00000100:00100000:3.0:1789760775.655786:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484325248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760775.655828:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24190976, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:3.0:1789760775.655841:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.655856:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=764s v=(0 0) 00010000:00010000:3.0:1789760775.655860:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760775.655867:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24190976->24195071] (req 24190976->24195071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.655888:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24190976->24195071] (req 24190976->24195071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760775.703498:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973224 is committed 00002000:00010000:3.0:1789760775.703863:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24190976 -> 24195072 00010000:00010000:3.0:1789760775.703878:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24190976->24195071] (req 24190976->24195071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.703890:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24190976->24195071] (req 24190976->24195071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.703899:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24190976->24195071] (req 24190976->24195071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.703907:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760775.703912:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24190976->24195071] (req 24190976->24195071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.703922:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24190976->24195071] (req 24190976->24195071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.703942:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24190976->24195071] (req 24190976->24195071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.703950:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b3b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24190976->24195071] (req 24190976->24195071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760775.704041:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8910380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484325248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48252us (48627us total) trans 4294973224 rc 0/0 00000100:00100000:3.0:1789760775.704051:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5935 00000100:00100000:3.0:1789760775.704107:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764952090332 time 764986705260 state 0x2 ref 0 00000100:00100000:1.0:1789760775.708198:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.708374:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484325376 00000100:00100000:2.0:1789760775.708384:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484325376 00000100:00100000:2.0:1789760775.708442:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764952090332 time 764986705260 state 0x0 ref 1 00000100:00100000:2.0:1789760775.708458:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764986965676 time 764986705260 state 0x1 ref 1 00000100:00100000:2.0:1789760775.708491:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764986965676 time 765039421714 state 0x0 ref 1 00000100:00100000:2.0:1789760775.708504:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484325376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.708566:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24195072, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:2.0:1789760775.708575:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.708589:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:2.0:1789760775.708594:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.708600:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24195072->24199167] (req 24195072->24199167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.708613:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24195072->24199167] (req 24195072->24199167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760775.752108:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973225 is committed 00002000:00010000:3.0:1789760775.752488:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24195072 -> 24199168 00010000:00010000:3.0:1789760775.752803:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24195072->24199167] (req 24195072->24199167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.752818:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24195072->24199167] (req 24195072->24199167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.752826:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24195072->24199167] (req 24195072->24199167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.752833:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760775.752837:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24195072->24199167] (req 24195072->24199167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.752938:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24195072->24199167] (req 24195072->24199167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.752962:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24195072->24199167] (req 24195072->24199167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.752973:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209b42 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24195072->24199167] (req 24195072->24199167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760775.753117:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484325376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44609us (44921us total) trans 4294973225 rc 0/0 00000100:00100000:3.0:1789760775.753128:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5936 00000100:00100000:3.0:1789760775.753139:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764986965676 time 765039421714 state 0x2 ref 0 00000100:00100000:1.0:1789760775.756527:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.756742:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484325504 00000100:00100000:2.0:1789760775.756752:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484325504 00000100:00100000:2.0:1789760775.756807:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 764986965676 time 765039421714 state 0x0 ref 1 00000100:00100000:2.0:1789760775.756821:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765039682130 time 765039421714 state 0x1 ref 1 00000100:00100000:2.0:1789760775.756847:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765039682130 time 765087778269 state 0x0 ref 1 00000100:00100000:2.0:1789760775.756860:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484325504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.756908:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24199168, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:2.0:1789760775.756917:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.756933:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:2.0:1789760775.756937:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.756944:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24199168->24203263] (req 24199168->24203263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.756958:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24199168->24203263] (req 24199168->24203263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760775.785448:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973226 is committed 00002000:00010000:3.0:1789760775.785569:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24199168 -> 24203264 00010000:00010000:3.0:1789760775.785584:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24199168->24203263] (req 24199168->24203263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.785597:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24199168->24203263] (req 24199168->24203263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.785605:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24199168->24203263] (req 24199168->24203263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.785612:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760775.785617:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24199168->24203263] (req 24199168->24203263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.785627:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24199168->24203263] (req 24199168->24203263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.785752:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24199168->24203263] (req 24199168->24203263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.785760:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209b49 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24199168->24203263] (req 24199168->24203263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760775.785927:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484325504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28998us (29340us total) trans 4294973226 rc 0/0 00000100:00100000:3.0:1789760775.785939:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5937 00000100:00100000:3.0:1789760775.785949:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765039682130 time 765087778269 state 0x2 ref 0 00000100:00100000:0.0:1789760775.791954:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760775.792117:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484325632 00000100:00100000:3.0:1789760775.792128:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484325632 00000100:00100000:3.0:1789760775.792177:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765039682130 time 765087778269 state 0x0 ref 1 00000100:00100000:3.0:1789760775.792194:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765088038685 time 765087778269 state 0x1 ref 1 00000100:00100000:3.0:1789760775.792306:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765088038685 time 765123236099 state 0x0 ref 1 00000100:00100000:3.0:1789760775.792320:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484325632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760775.792366:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24203264, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:3.0:1789760775.792375:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.792392:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760775.792397:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760775.792403:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24203264->24207359] (req 24203264->24207359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.792417:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24203264->24207359] (req 24203264->24207359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760775.832271:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24203264 -> 24207360 00010000:00010000:3.0:1789760775.832292:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24203264->24207359] (req 24203264->24207359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.832305:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24203264->24207359] (req 24203264->24207359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.832313:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24203264->24207359] (req 24203264->24207359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.832320:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760775.832325:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24203264->24207359] (req 24203264->24207359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.832334:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24203264->24207359] (req 24203264->24207359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.832353:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24203264->24207359] (req 24203264->24207359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.832360:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b50 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24203264->24207359] (req 24203264->24207359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760775.832554:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973227 is committed 00000100:00100000:3.0:1789760775.832629:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8912300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484325632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40306us (40680us total) trans 4294973227 rc 0/0 00000100:00100000:3.0:1789760775.832650:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5938 00000100:00100000:3.0:1789760775.832660:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765088038685 time 765123236099 state 0x2 ref 0 00000100:00100000:1.0:1789760775.840699:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760775.840804:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484325760 00000100:00100000:3.0:1789760775.840813:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484325760 00000100:00100000:3.0:1789760775.840925:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765088038685 time 765123236099 state 0x0 ref 1 00000100:00100000:3.0:1789760775.840940:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765123496515 time 765123236099 state 0x1 ref 1 00000100:00100000:3.0:1789760775.841020:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765123496515 time 765171950788 state 0x0 ref 1 00000100:00100000:3.0:1789760775.841033:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484325760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760775.841120:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24207360, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:3.0:1789760775.841129:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.841156:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760775.841161:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760775.841167:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24207360->24211455] (req 24207360->24211455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.841181:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24207360->24211455] (req 24207360->24211455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760775.887015:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973228 is committed 00002000:00010000:2.0:1789760775.887192:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24207360 -> 24211456 00010000:00010000:2.0:1789760775.887210:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24207360->24211455] (req 24207360->24211455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.887223:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24207360->24211455] (req 24207360->24211455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.887232:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24207360->24211455] (req 24207360->24211455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.887239:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760775.887244:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24207360->24211455] (req 24207360->24211455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.887254:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24207360->24211455] (req 24207360->24211455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.887278:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24207360->24211455] (req 24207360->24211455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.887286:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209b57 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24207360->24211455] (req 24207360->24211455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760775.887533:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484325760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46497us (46839us total) trans 4294973228 rc 0/0 00000100:00100000:2.0:1789760775.887544:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5939 00000100:00100000:2.0:1789760775.887555:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765123496515 time 765171950788 state 0x2 ref 0 00000100:00100000:0.0:1789760775.891439:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.891526:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484325888 00000100:00100000:2.0:1789760775.891536:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484325888 00000100:00100000:2.0:1789760775.891589:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765123496515 time 765171950788 state 0x0 ref 1 00000100:00100000:2.0:1789760775.891604:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765172211204 time 765171950788 state 0x1 ref 1 00000100:00100000:2.0:1789760775.891677:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765172211204 time 765222606944 state 0x0 ref 1 00000100:00100000:2.0:1789760775.891690:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484325888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.891751:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24211456, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:2.0:1789760775.891761:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.891776:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:2.0:1789760775.891780:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.891786:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24211456->24215551] (req 24211456->24215551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.891799:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24211456->24215551] (req 24211456->24215551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760775.921872:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973229 is committed 00002000:00010000:0.0:1789760775.922045:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24211456 -> 24215552 00010000:00010000:0.0:1789760775.922214:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24211456->24215551] (req 24211456->24215551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.922230:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24211456->24215551] (req 24211456->24215551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.922239:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24211456->24215551] (req 24211456->24215551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.922246:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760775.922250:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24211456->24215551] (req 24211456->24215551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.922260:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24211456->24215551] (req 24211456->24215551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.922281:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24211456->24215551] (req 24211456->24215551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.922288:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf3826396209b5e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24211456->24215551] (req 24211456->24215551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760775.922372:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484325888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30675us (30933us total) trans 4294973229 rc 0/0 00000100:00100000:0.0:1789760775.922382:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5940 00000100:00100000:0.0:1789760775.922392:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765172211204 time 765222606944 state 0x2 ref 0 00000100:00100000:0.0:1789760775.928618:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760775.928856:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484326016 00000100:00100000:2.0:1789760775.928867:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484326016 00000100:00100000:2.0:1789760775.928922:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765172211204 time 765222606944 state 0x0 ref 1 00000100:00100000:2.0:1789760775.928938:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765222867360 time 765222606944 state 0x1 ref 1 00000100:00100000:2.0:1789760775.928971:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765222867360 time 765259901891 state 0x0 ref 1 00000100:00100000:2.0:1789760775.928991:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484326016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760775.929040:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24215552, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:2.0:1789760775.929050:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.929115:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:2.0:1789760775.929120:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760775.929128:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24215552->24219647] (req 24215552->24219647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760775.929144:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24215552->24219647] (req 24215552->24219647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760775.992275:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973230 is committed 00002000:00010000:0.0:1789760775.992483:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24215552 -> 24219648 00010000:00010000:0.0:1789760775.992499:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24215552->24219647] (req 24215552->24219647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.992511:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24215552->24219647] (req 24215552->24219647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.992519:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24215552->24219647] (req 24215552->24219647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.992527:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760775.992531:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24215552->24219647] (req 24215552->24219647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.992546:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24215552->24219647] (req 24215552->24219647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.992564:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24215552->24219647] (req 24215552->24219647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760775.992572:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209b65 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24215552->24219647] (req 24215552->24219647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760775.992661:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484326016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63675us (64049us total) trans 4294973230 rc 0/0 00000100:00100000:0.0:1789760775.992671:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5941 00000100:00100000:0.0:1789760775.992680:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765222867360 time 765259901891 state 0x2 ref 0 00000100:00100000:1.0:1789760775.999257:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760775.999354:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484326144 00000100:00100000:3.0:1789760775.999365:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484326144 00000100:00100000:3.0:1789760775.999421:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765222867360 time 765259901891 state 0x0 ref 1 00000100:00100000:3.0:1789760775.999437:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765260162307 time 765259901891 state 0x1 ref 1 00000100:00100000:3.0:1789760775.999469:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765260162307 time 765330399925 state 0x0 ref 1 00000100:00100000:3.0:1789760775.999481:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484326144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760775.999532:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24219648, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:3.0:1789760775.999540:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.999556:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760775.999561:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760775.999568:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24219648->24223743] (req 24219648->24223743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760775.999582:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24219648->24223743] (req 24219648->24223743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760776.057391:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973231 is committed 00002000:00010000:3.0:1789760776.057498:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24219648 -> 24223744 00010000:00010000:3.0:1789760776.057514:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24219648->24223743] (req 24219648->24223743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.057531:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24219648->24223743] (req 24219648->24223743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.057540:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24219648->24223743] (req 24219648->24223743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.057548:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760776.057553:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24219648->24223743] (req 24219648->24223743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.057563:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24219648->24223743] (req 24219648->24223743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.057582:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24219648->24223743] (req 24219648->24223743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.057590:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b6c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24219648->24223743] (req 24219648->24223743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760776.057680:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484326144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58196us (58428us total) trans 4294973231 rc 0/0 00000100:00100000:3.0:1789760776.057689:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5942 00000100:00100000:3.0:1789760776.057698:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765260162307 time 765330399925 state 0x2 ref 0 00000100:00100000:1.0:1789760776.062616:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760776.062867:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484326272 00000100:00100000:3.0:1789760776.062880:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484326272 00000100:00100000:3.0:1789760776.062951:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765260162307 time 765330399925 state 0x0 ref 1 00000100:00100000:3.0:1789760776.062968:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765330660341 time 765330399925 state 0x1 ref 1 00000100:00100000:3.0:1789760776.063004:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765330660341 time 765393934517 state 0x0 ref 1 00000100:00100000:3.0:1789760776.063017:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484326272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.063104:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24223744, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:3.0:1789760776.063114:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.063131:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760776.063136:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.063142:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24223744->24227839] (req 24223744->24227839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.063155:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24223744->24227839] (req 24223744->24227839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760776.063801:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00010000:00010000:1.0:1789760776.081710:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760776.097099:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973232 is committed 00002000:00010000:3.0:1789760776.097275:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24223744 -> 24227840 00010000:00010000:3.0:1789760776.097290:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24223744->24227839] (req 24223744->24227839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.097304:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24223744->24227839] (req 24223744->24227839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.097320:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24223744->24227839] (req 24223744->24227839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.097328:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760776.097333:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24223744->24227839] (req 24223744->24227839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.097343:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24223744->24227839] (req 24223744->24227839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.097362:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24223744->24227839] (req 24223744->24227839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.097371:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b73 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24223744->24227839] (req 24223744->24227839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760776.097462:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484326272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34443us (34851us total) trans 4294973232 rc 0/0 00000100:00100000:3.0:1789760776.097472:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5943 00000100:00100000:3.0:1789760776.097482:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765330660341 time 765393934517 state 0x2 ref 0 00000100:00100000:0.0:1789760776.101319:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760776.101424:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484326400 00000100:00100000:2.0:1789760776.101434:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484326400 00000100:00100000:2.0:1789760776.101490:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765330660341 time 765393934517 state 0x0 ref 1 00000100:00100000:2.0:1789760776.101509:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765394194933 time 765393934517 state 0x1 ref 1 00000100:00100000:2.0:1789760776.101540:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765394194933 time 765432470530 state 0x0 ref 1 00000100:00100000:2.0:1789760776.101554:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484326400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760776.101604:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24227840, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:2.0:1789760776.101613:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.101628:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:2.0:1789760776.101633:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760776.101639:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24227840->24231935] (req 24227840->24231935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.101654:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24227840->24231935] (req 24227840->24231935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760776.136004:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973233 is committed 00002000:00010000:2.0:1789760776.136194:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24227840 -> 24231936 00010000:00010000:2.0:1789760776.136211:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24227840->24231935] (req 24227840->24231935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.136224:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24227840->24231935] (req 24227840->24231935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.136234:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24227840->24231935] (req 24227840->24231935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.136246:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760776.136251:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24227840->24231935] (req 24227840->24231935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.136260:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24227840->24231935] (req 24227840->24231935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.136279:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24227840->24231935] (req 24227840->24231935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.136287:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209b7a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24227840->24231935] (req 24227840->24231935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760776.136368:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484326400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34813us (35054us total) trans 4294973233 rc 0/0 00000100:00100000:2.0:1789760776.136378:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5944 00000100:00100000:2.0:1789760776.136388:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765394194933 time 765432470530 state 0x2 ref 0 00000100:00100000:1.0:1789760776.140217:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760776.140442:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484326528 00000100:00100000:0.0:1789760776.140451:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484326528 00000100:00100000:0.0:1789760776.140520:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765394194933 time 765432470530 state 0x0 ref 1 00000100:00100000:0.0:1789760776.140537:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765432730946 time 765432470530 state 0x1 ref 1 00000100:00100000:0.0:1789760776.140568:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765432730946 time 765471498572 state 0x0 ref 1 00000100:00100000:0.0:1789760776.140582:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484326528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760776.140633:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24231936, mtime: 0x6aad9507, atime: 0x0, ctime: 0x6aad9507, blocks: 0xc007 00010000:00010000:0.0:1789760776.140641:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.140658:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:0.0:1789760776.140662:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760776.140668:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24231936->24236031] (req 24231936->24236031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.140682:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24231936->24236031] (req 24231936->24236031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760776.174124:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24231936 -> 24236032 00000001:00080000:1.0:1789760776.174124:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973234 is committed 00002000:00010000:0.0:1789760776.174131:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760775 -> 1789760776 00002000:00010000:0.0:1789760776.174134:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760775 -> 1789760776 00010000:00010000:0.0:1789760776.174145:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24231936->24236031] (req 24231936->24236031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.174158:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24231936->24236031] (req 24231936->24236031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.174167:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24231936->24236031] (req 24231936->24236031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.174174:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760776.174179:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24231936->24236031] (req 24231936->24236031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.174192:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24231936->24236031] (req 24231936->24236031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.174211:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24231936->24236031] (req 24231936->24236031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.174216:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209b81 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24231936->24236031] (req 24231936->24236031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760776.174341:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484326528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33757us (34126us total) trans 4294973234 rc 0/0 00000100:00100000:2.0:1789760776.174357:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5945 00000100:00100000:2.0:1789760776.174367:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765432730946 time 765471498572 state 0x2 ref 0 00000100:00100000:2.0:1789760776.180372:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760776.180555:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484326656 00000100:00100000:3.0:1789760776.180566:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484326656 00000100:00100000:3.0:1789760776.180623:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765432730946 time 765471498572 state 0x0 ref 1 00000100:00100000:3.0:1789760776.180639:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765471758988 time 765471498572 state 0x1 ref 1 00000100:00100000:3.0:1789760776.180670:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765471758988 time 765511601459 state 0x0 ref 1 00000100:00100000:3.0:1789760776.180683:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484326656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.180744:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24236032, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760776.180753:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.180769:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760776.180776:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.180782:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24236032->24240127] (req 24236032->24240127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.180797:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24236032->24240127] (req 24236032->24240127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760776.273454:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973235 is committed 00002000:00010000:2.0:1789760776.273548:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24236032 -> 24240128 00010000:00010000:2.0:1789760776.273563:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24236032->24240127] (req 24236032->24240127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.273703:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24236032->24240127] (req 24236032->24240127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.273712:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24236032->24240127] (req 24236032->24240127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.273783:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760776.273790:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24236032->24240127] (req 24236032->24240127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.273800:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24236032->24240127] (req 24236032->24240127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.273819:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24236032->24240127] (req 24236032->24240127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.273828:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209b88 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24236032->24240127] (req 24236032->24240127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760776.273981:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484326656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 93295us (93615us total) trans 4294973235 rc 0/0 00000100:00100000:2.0:1789760776.273993:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5946 00000100:00100000:2.0:1789760776.274003:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765471758988 time 765511601459 state 0x2 ref 0 00000100:00100000:0.0:1789760776.278995:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760776.283025:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484326784 00000100:00100000:3.0:1789760776.283040:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484326784 00000100:00100000:3.0:1789760776.283157:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765471758988 time 765511601459 state 0x0 ref 1 00000100:00100000:3.0:1789760776.283173:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765511861875 time 765511601459 state 0x1 ref 1 00000100:00100000:3.0:1789760776.283197:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765511861875 time 765614128839 state 0x0 ref 1 00000100:00100000:3.0:1789760776.283211:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484326784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.283266:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24240128, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760776.283276:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.283628:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760776.283634:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.283641:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24240128->24244223] (req 24240128->24244223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.283656:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24240128->24244223] (req 24240128->24244223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760776.336527:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973236 is committed 00002000:00010000:3.0:1789760776.336644:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24240128 -> 24244224 00010000:00010000:3.0:1789760776.336661:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24240128->24244223] (req 24240128->24244223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.336674:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24240128->24244223] (req 24240128->24244223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.336683:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24240128->24244223] (req 24240128->24244223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.336691:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760776.336695:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24240128->24244223] (req 24240128->24244223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.336705:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24240128->24244223] (req 24240128->24244223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.336801:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24240128->24244223] (req 24240128->24244223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.336810:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209b8f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24240128->24244223] (req 24240128->24244223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760776.336915:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484326784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53702us (57924us total) trans 4294973236 rc 0/0 00000100:00100000:3.0:1789760776.336926:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5947 00000100:00100000:3.0:1789760776.336934:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765511861875 time 765614128839 state 0x2 ref 0 00000100:00100000:1.0:1789760776.339737:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760776.339909:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484326912 00000100:00100000:2.0:1789760776.339919:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484326912 00000100:00100000:2.0:1789760776.340031:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765511861875 time 765614128839 state 0x0 ref 1 00000100:00100000:2.0:1789760776.340049:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765614389255 time 765614128839 state 0x1 ref 1 00000100:00100000:3.0:1789760776.340198:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765614389255 time 765671126643 state 0x0 ref 1 00000100:00100000:3.0:1789760776.340217:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484326912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.340264:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24244224, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760776.340273:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.340346:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760776.340352:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.340359:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24244224->24248319] (req 24244224->24248319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.340374:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24244224->24248319] (req 24244224->24248319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760776.372320:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973237 is committed 00002000:00010000:2.0:1789760776.372373:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24244224 -> 24248320 00010000:00010000:2.0:1789760776.372388:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24244224->24248319] (req 24244224->24248319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.372401:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24244224->24248319] (req 24244224->24248319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.372411:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24244224->24248319] (req 24244224->24248319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.372419:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760776.372424:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24244224->24248319] (req 24244224->24248319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.372439:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24244224->24248319] (req 24244224->24248319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.372458:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24244224->24248319] (req 24244224->24248319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.372472:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209b96 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24244224->24248319] (req 24244224->24248319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760776.372601:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484326912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32383us (32871us total) trans 4294973237 rc 0/0 00000100:00100000:2.0:1789760776.372611:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5948 00000100:00100000:2.0:1789760776.372620:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765614389255 time 765671126643 state 0x2 ref 0 00000100:00100000:2.0:1789760776.376125:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760776.376258:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484327040 00000100:00100000:1.0:1789760776.376267:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484327040 00000100:00100000:1.0:1789760776.376319:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765614389255 time 765671126643 state 0x0 ref 1 00000100:00100000:1.0:1789760776.376334:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765671387059 time 765671126643 state 0x1 ref 1 00000100:00100000:1.0:1789760776.376395:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765671387059 time 765707324571 state 0x0 ref 1 00000100:00100000:1.0:1789760776.376412:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484327040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760776.376458:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24248320, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:1.0:1789760776.376466:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.376483:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:1.0:1789760776.376488:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760776.376494:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24248320->24252415] (req 24248320->24252415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.376507:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24248320->24252415] (req 24248320->24252415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760776.401507:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973238 is committed 00002000:00010000:2.0:1789760776.401614:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24248320 -> 24252416 00010000:00010000:2.0:1789760776.401629:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24248320->24252415] (req 24248320->24252415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.401643:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24248320->24252415] (req 24248320->24252415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.401652:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24248320->24252415] (req 24248320->24252415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.401660:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760776.401665:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24248320->24252415] (req 24248320->24252415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.401674:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24248320->24252415] (req 24248320->24252415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.401698:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24248320->24252415] (req 24248320->24252415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.401705:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209b9d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24248320->24252415] (req 24248320->24252415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760776.401831:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484327040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25418us (25710us total) trans 4294973238 rc 0/0 00000100:00100000:2.0:1789760776.401850:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5949 00000100:00100000:2.0:1789760776.401861:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765671387059 time 765707324571 state 0x2 ref 0 00000100:00100000:0.0:1789760776.405346:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760776.405437:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484327168 00000100:00100000:2.0:1789760776.405447:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484327168 00000100:00100000:2.0:1789760776.405503:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765671387059 time 765707324571 state 0x0 ref 1 00000100:00100000:2.0:1789760776.405518:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765707584987 time 765707324571 state 0x1 ref 1 00000100:00100000:2.0:1789760776.405581:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765707584987 time 765736511269 state 0x0 ref 1 00000100:00100000:2.0:1789760776.405593:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484327168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760776.405636:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24252416, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:2.0:1789760776.405644:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.405659:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:2.0:1789760776.405664:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760776.405670:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24252416->24256511] (req 24252416->24256511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.405682:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24252416->24256511] (req 24252416->24256511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760776.444029:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973239 is committed 00002000:00010000:2.0:1789760776.444327:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24252416 -> 24256512 00010000:00010000:2.0:1789760776.444343:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24252416->24256511] (req 24252416->24256511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.444357:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24252416->24256511] (req 24252416->24256511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.444367:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24252416->24256511] (req 24252416->24256511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.444375:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760776.444381:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24252416->24256511] (req 24252416->24256511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.444391:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24252416->24256511] (req 24252416->24256511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.444412:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24252416->24256511] (req 24252416->24256511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.444426:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209ba4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24252416->24256511] (req 24252416->24256511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760776.444573:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484327168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38978us (39232us total) trans 4294973239 rc 0/0 00000100:00100000:2.0:1789760776.444583:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5950 00000100:00100000:2.0:1789760776.444592:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765707584987 time 765736511269 state 0x2 ref 0 00000100:00100000:0.0:1789760776.447444:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760776.447522:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484327296 00000100:00100000:3.0:1789760776.447532:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484327296 00000100:00100000:3.0:1789760776.447584:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765707584987 time 765736511269 state 0x0 ref 1 00000100:00100000:3.0:1789760776.447599:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765736771685 time 765736511269 state 0x1 ref 1 00000100:00100000:3.0:1789760776.447659:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765736771685 time 765778589420 state 0x0 ref 1 00000100:00100000:3.0:1789760776.447673:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484327296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.447718:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24256512, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760776.447728:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.447743:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760776.447747:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.447753:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24256512->24260607] (req 24256512->24260607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.447767:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24256512->24260607] (req 24256512->24260607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760776.489440:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973240 is committed 00002000:00010000:3.0:1789760776.489548:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24256512 -> 24260608 00010000:00010000:3.0:1789760776.489564:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24256512->24260607] (req 24256512->24260607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.489584:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24256512->24260607] (req 24256512->24260607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.489593:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24256512->24260607] (req 24256512->24260607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.489601:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760776.489605:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24256512->24260607] (req 24256512->24260607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.489615:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24256512->24260607] (req 24256512->24260607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.489637:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24256512->24260607] (req 24256512->24260607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.489644:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bab lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24256512->24260607] (req 24256512->24260607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760776.489763:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484327296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42090us (42323us total) trans 4294973240 rc 0/0 00000100:00100000:3.0:1789760776.489773:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5951 00000100:00100000:3.0:1789760776.489782:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765736771685 time 765778589420 state 0x2 ref 0 00000100:00100000:0.0:1789760776.492905:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760776.492987:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484327424 00000100:00100000:3.0:1789760776.492997:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484327424 00000100:00100000:3.0:1789760776.493047:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765736771685 time 765778589420 state 0x0 ref 1 00000100:00100000:3.0:1789760776.493135:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765778849836 time 765778589420 state 0x1 ref 1 00000100:00100000:3.0:1789760776.493261:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765778849836 time 765824190887 state 0x0 ref 1 00000100:00100000:3.0:1789760776.493274:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484327424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.493322:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24260608, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760776.493330:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.493346:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760776.493350:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.493355:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24260608->24264703] (req 24260608->24264703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.493369:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24260608->24264703] (req 24260608->24264703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760776.516403:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973241 is committed 00002000:00010000:0.0:1789760776.516516:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24260608 -> 24264704 00010000:00010000:0.0:1789760776.516532:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24260608->24264703] (req 24260608->24264703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.516544:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24260608->24264703] (req 24260608->24264703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.516553:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24260608->24264703] (req 24260608->24264703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.516567:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760776.516572:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24260608->24264703] (req 24260608->24264703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.516580:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24260608->24264703] (req 24260608->24264703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.516600:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24260608->24264703] (req 24260608->24264703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.516608:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209bb2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24260608->24264703] (req 24260608->24264703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760776.516698:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484327424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23421us (23798us total) trans 4294973241 rc 0/0 00000100:00100000:0.0:1789760776.516708:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5952 00000100:00100000:0.0:1789760776.516717:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765778849836 time 765824190887 state 0x2 ref 0 00000100:00100000:0.0:1789760776.521580:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760776.522097:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484327552 00000100:00100000:2.0:1789760776.522108:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484327552 00000100:00100000:2.0:1789760776.522168:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765778849836 time 765824190887 state 0x0 ref 1 00000100:00100000:2.0:1789760776.522183:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765824451303 time 765824190887 state 0x1 ref 1 00000100:00100000:2.0:1789760776.522209:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765824451303 time 765853140729 state 0x0 ref 1 00000100:00100000:2.0:1789760776.522222:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484327552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760776.522269:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24264704, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:2.0:1789760776.522278:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.522293:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:2.0:1789760776.522297:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760776.522303:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24264704->24268799] (req 24264704->24268799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.522321:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24264704->24268799] (req 24264704->24268799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760776.548108:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973242 is committed 00002000:00010000:0.0:1789760776.548203:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24264704 -> 24268800 00010000:00010000:0.0:1789760776.548218:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24264704->24268799] (req 24264704->24268799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.548231:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24264704->24268799] (req 24264704->24268799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.548247:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24264704->24268799] (req 24264704->24268799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.548254:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760776.548259:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24264704->24268799] (req 24264704->24268799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.548268:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24264704->24268799] (req 24264704->24268799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.548289:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24264704->24268799] (req 24264704->24268799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.548296:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209bb9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24264704->24268799] (req 24264704->24268799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760776.548385:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484327552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26162us (26810us total) trans 4294973242 rc 0/0 00000100:00100000:0.0:1789760776.548395:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5953 00000100:00100000:0.0:1789760776.548405:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765824451303 time 765853140729 state 0x2 ref 0 00000100:00100000:1.0:1789760776.551585:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760776.551700:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484327680 00000100:00100000:2.0:1789760776.551709:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484327680 00000100:00100000:2.0:1789760776.551763:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765824451303 time 765853140729 state 0x0 ref 1 00000100:00100000:2.0:1789760776.551778:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765853401145 time 765853140729 state 0x1 ref 1 00000100:00100000:2.0:1789760776.551807:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765853401145 time 765882738394 state 0x0 ref 1 00000100:00100000:2.0:1789760776.551820:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484327680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760776.551880:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24268800, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:2.0:1789760776.551889:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.551904:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:2.0:1789760776.551908:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760776.551914:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24268800->24272895] (req 24268800->24272895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.551926:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24268800->24272895] (req 24268800->24272895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760776.578134:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973243 is committed 00002000:00010000:2.0:1789760776.578342:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24268800 -> 24272896 00010000:00010000:2.0:1789760776.578362:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24268800->24272895] (req 24268800->24272895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.578377:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24268800->24272895] (req 24268800->24272895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.578386:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24268800->24272895] (req 24268800->24272895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.578393:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760776.578399:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24268800->24272895] (req 24268800->24272895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.578407:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24268800->24272895] (req 24268800->24272895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.578429:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24268800->24272895] (req 24268800->24272895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.578436:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209bc0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24268800->24272895] (req 24268800->24272895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760776.578526:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484327680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26704us (26945us total) trans 4294973243 rc 0/0 00000100:00100000:2.0:1789760776.578536:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5954 00000100:00100000:2.0:1789760776.578547:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765853401145 time 765882738394 state 0x2 ref 0 00000100:00100000:1.0:1789760776.582388:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760776.582475:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484327808 00000100:00100000:3.0:1789760776.582485:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484327808 00000100:00100000:3.0:1789760776.582538:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765853401145 time 765882738394 state 0x0 ref 1 00000100:00100000:3.0:1789760776.582553:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765882998810 time 765882738394 state 0x1 ref 1 00000100:00100000:3.0:1789760776.582620:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765882998810 time 765913550036 state 0x0 ref 1 00000100:00100000:3.0:1789760776.582637:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484327808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.582687:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24272896, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760776.582696:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.582712:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760776.582717:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.582724:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24272896->24276991] (req 24272896->24276991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.582737:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24272896->24276991] (req 24272896->24276991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760776.620300:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973244 is committed 00002000:00010000:0.0:1789760776.620413:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24272896 -> 24276992 00010000:00010000:0.0:1789760776.620428:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24272896->24276991] (req 24272896->24276991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.620447:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24272896->24276991] (req 24272896->24276991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.620456:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24272896->24276991] (req 24272896->24276991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.620464:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760776.620468:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24272896->24276991] (req 24272896->24276991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.620477:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24272896->24276991] (req 24272896->24276991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.620497:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24272896->24276991] (req 24272896->24276991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.620505:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209bc7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24272896->24276991] (req 24272896->24276991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760776.620596:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484327808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37957us (38214us total) trans 4294973244 rc 0/0 00000100:00100000:0.0:1789760776.620606:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5955 00000100:00100000:0.0:1789760776.620616:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765882998810 time 765913550036 state 0x2 ref 0 00000100:00100000:0.0:1789760776.624577:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760776.624674:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484327936 00000100:00100000:3.0:1789760776.624684:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484327936 00000100:00100000:3.0:1789760776.624738:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765882998810 time 765913550036 state 0x0 ref 1 00000100:00100000:3.0:1789760776.624754:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765913810452 time 765913550036 state 0x1 ref 1 00000100:00100000:3.0:1789760776.624826:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765913810452 time 765955756554 state 0x0 ref 1 00000100:00100000:3.0:1789760776.624850:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484327936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.624905:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24276992, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760776.624915:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.624937:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:3.0:1789760776.624941:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.624948:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24276992->24281087] (req 24276992->24281087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.624963:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24276992->24281087] (req 24276992->24281087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760776.650293:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973245 is committed 00002000:00010000:3.0:1789760776.650371:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24276992 -> 24281088 00010000:00010000:3.0:1789760776.650387:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24276992->24281087] (req 24276992->24281087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.650399:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24276992->24281087] (req 24276992->24281087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.650409:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24276992->24281087] (req 24276992->24281087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.650416:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760776.650421:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24276992->24281087] (req 24276992->24281087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.650429:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24276992->24281087] (req 24276992->24281087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.650448:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24276992->24281087] (req 24276992->24281087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.650457:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bce lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24276992->24281087] (req 24276992->24281087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760776.650555:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484327936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25712us (25982us total) trans 4294973245 rc 0/0 00000100:00100000:3.0:1789760776.650566:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5956 00000100:00100000:3.0:1789760776.650577:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765913810452 time 765955756554 state 0x2 ref 0 00000100:00100000:2.0:1789760776.653486:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760776.653564:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484328064 00000100:00100000:0.0:1789760776.653573:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484328064 00000100:00100000:0.0:1789760776.653632:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765913810452 time 765955756554 state 0x0 ref 1 00000100:00100000:0.0:1789760776.653647:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765956016970 time 765955756554 state 0x1 ref 1 00000100:00100000:0.0:1789760776.653678:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765956016970 time 765984609378 state 0x0 ref 1 00000100:00100000:0.0:1789760776.653693:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484328064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760776.653743:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24281088, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:0.0:1789760776.653751:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.653766:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=765s v=(0 0) 00010000:00010000:0.0:1789760776.653770:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760776.653777:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24281088->24285183] (req 24281088->24285183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.653791:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24281088->24285183] (req 24281088->24285183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760776.687133:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973246 is committed 00002000:00010000:0.0:1789760776.687225:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24281088 -> 24285184 00010000:00010000:0.0:1789760776.687238:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24281088->24285183] (req 24281088->24285183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.687247:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24281088->24285183] (req 24281088->24285183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.687258:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24281088->24285183] (req 24281088->24285183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.687263:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760776.687267:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24281088->24285183] (req 24281088->24285183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.687274:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24281088->24285183] (req 24281088->24285183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.687295:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24281088->24285183] (req 24281088->24285183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.687302:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bd5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24281088->24285183] (req 24281088->24285183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760776.687443:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484328064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33747us (33958us total) trans 4294973246 rc 0/0 00000100:00100000:0.0:1789760776.687453:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5957 00000100:00100000:0.0:1789760776.687463:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765956016970 time 765984609378 state 0x2 ref 0 00000100:00100000:1.0:1789760776.690220:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760776.690339:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484328192 00000100:00100000:2.0:1789760776.690349:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484328192 00000100:00100000:2.0:1789760776.690401:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765956016970 time 765984609378 state 0x0 ref 1 00000100:00100000:2.0:1789760776.690416:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765984869794 time 765984609378 state 0x1 ref 1 00000100:00100000:2.0:1789760776.690446:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765984869794 time 766021376989 state 0x0 ref 1 00000100:00100000:2.0:1789760776.690459:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484328192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760776.690513:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24285184, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:2.0:1789760776.690522:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.690537:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:2.0:1789760776.690542:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760776.690548:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24285184->24289279] (req 24285184->24289279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.690562:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24285184->24289279] (req 24285184->24289279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760776.722684:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973247 is committed 00002000:00010000:1.0:1789760776.722758:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24285184 -> 24289280 00010000:00010000:1.0:1789760776.722774:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24285184->24289279] (req 24285184->24289279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.722787:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24285184->24289279] (req 24285184->24289279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.722797:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24285184->24289279] (req 24285184->24289279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.722805:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760776.722810:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24285184->24289279] (req 24285184->24289279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.722819:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24285184->24289279] (req 24285184->24289279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.722837:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24285184->24289279] (req 24285184->24289279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.722845:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209bdc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24285184->24289279] (req 24285184->24289279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760776.722933:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484328192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32472us (32718us total) trans 4294973247 rc 0/0 00000100:00100000:1.0:1789760776.722943:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5958 00000100:00100000:1.0:1789760776.722954:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765984869794 time 766021376989 state 0x2 ref 0 00000100:00100000:3.0:1789760776.728755:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760776.728995:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484328320 00000100:00100000:0.0:1789760776.729004:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484328320 00000100:00100000:0.0:1789760776.729095:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 765984869794 time 766021376989 state 0x0 ref 1 00000100:00100000:0.0:1789760776.729112:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766021637405 time 766021376989 state 0x1 ref 1 00000100:00100000:2.0:1789760776.729230:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766021637405 time 766060158448 state 0x0 ref 1 00000100:00100000:2.0:1789760776.729286:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484328320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760776.729335:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24289280, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:2.0:1789760776.729344:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.729359:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:2.0:1789760776.729363:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760776.729369:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24289280->24293375] (req 24289280->24293375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760776.729383:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24289280->24293375] (req 24289280->24293375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760776.765154:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24289280 -> 24293376 00010000:00010000:1.0:1789760776.765175:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24289280->24293375] (req 24289280->24293375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.765194:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24289280->24293375] (req 24289280->24293375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.765204:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24289280->24293375] (req 24289280->24293375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.765211:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760776.765216:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24289280->24293375] (req 24289280->24293375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.765225:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24289280->24293375] (req 24289280->24293375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.765244:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24289280->24293375] (req 24289280->24293375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.765252:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf3826396209be3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24289280->24293375] (req 24289280->24293375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760776.765518:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484328320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36229us (36766us total) trans 4294973248 rc 0/0 00000100:00100000:1.0:1789760776.765528:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5959 00000100:00100000:1.0:1789760776.765537:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766021637405 time 766060158448 state 0x2 ref 0 00000001:00080000:1.0:1789760776.765587:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973248 is committed 00000100:00100000:2.0:1789760776.774681:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760776.774767:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484328448 00000100:00100000:3.0:1789760776.774778:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484328448 00000100:00100000:3.0:1789760776.774832:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766021637405 time 766060158448 state 0x0 ref 1 00000100:00100000:3.0:1789760776.774847:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766060418864 time 766060158448 state 0x1 ref 1 00000100:00100000:3.0:1789760776.774880:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766060418864 time 766105810202 state 0x0 ref 1 00000100:00100000:3.0:1789760776.774893:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484328448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.774943:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24293376, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760776.774952:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.774969:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:3.0:1789760776.774974:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.774980:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24293376->24297471] (req 24293376->24297471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.774995:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24293376->24297471] (req 24293376->24297471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760776.821322:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24293376 -> 24297472 00000001:00080000:1.0:1789760776.821336:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973249 is committed 00010000:00010000:0.0:1789760776.821341:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24293376->24297471] (req 24293376->24297471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.821353:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24293376->24297471] (req 24293376->24297471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.821363:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24293376->24297471] (req 24293376->24297471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.821370:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760776.821375:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24293376->24297471] (req 24293376->24297471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.821383:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24293376->24297471] (req 24293376->24297471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.821464:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24293376->24297471] (req 24293376->24297471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.821473:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209bea lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24293376->24297471] (req 24293376->24297471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760776.821563:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484328448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46668us (46886us total) trans 4294973249 rc 0/0 00000100:00100000:0.0:1789760776.821573:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5960 00000100:00100000:0.0:1789760776.821583:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766060418864 time 766105810202 state 0x2 ref 0 00000100:00100000:3.0:1789760776.827593:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760776.827785:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484328576 00000100:00100000:0.0:1789760776.827801:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484328576 00000100:00100000:0.0:1789760776.827868:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766060418864 time 766105810202 state 0x0 ref 1 00000100:00100000:0.0:1789760776.827885:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766106070618 time 766105810202 state 0x1 ref 1 00000100:00100000:0.0:1789760776.827909:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766106070618 time 766158839969 state 0x0 ref 1 00000100:00100000:0.0:1789760776.827924:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484328576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760776.827976:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24297472, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:0.0:1789760776.827986:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.828002:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:0.0:1789760776.828007:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760776.828013:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24297472->24301567] (req 24297472->24301567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.828028:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24297472->24301567] (req 24297472->24301567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760776.869254:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973250 is committed 00002000:00010000:1.0:1789760776.869525:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24297472 -> 24301568 00010000:00010000:1.0:1789760776.869546:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24297472->24301567] (req 24297472->24301567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.869560:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24297472->24301567] (req 24297472->24301567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.869570:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24297472->24301567] (req 24297472->24301567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.869578:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760776.869583:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24297472->24301567] (req 24297472->24301567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.869593:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24297472->24301567] (req 24297472->24301567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.869613:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24297472->24301567] (req 24297472->24301567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.869622:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209bf1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24297472->24301567] (req 24297472->24301567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760776.869722:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484328576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41797us (42141us total) trans 4294973250 rc 0/0 00000100:00100000:1.0:1789760776.869734:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5961 00000100:00100000:1.0:1789760776.869745:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766106070618 time 766158839969 state 0x2 ref 0 00000100:00100000:3.0:1789760776.872633:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760776.872712:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484328704 00000100:00100000:1.0:1789760776.872728:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484328704 00000100:00100000:1.0:1789760776.872793:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766106070618 time 766158839969 state 0x0 ref 1 00000100:00100000:1.0:1789760776.872807:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766159100385 time 766158839969 state 0x1 ref 1 00000100:00100000:1.0:1789760776.872843:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766159100385 time 766203773809 state 0x0 ref 1 00000100:00100000:1.0:1789760776.872857:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484328704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760776.872902:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24301568, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:1.0:1789760776.872911:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.872927:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:1.0:1789760776.872932:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760776.872938:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24301568->24305663] (req 24301568->24305663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760776.872952:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24301568->24305663] (req 24301568->24305663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760776.909745:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973251 is committed 00002000:00010000:0.0:1789760776.910205:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24301568 -> 24305664 00010000:00010000:0.0:1789760776.910220:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24301568->24305663] (req 24301568->24305663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.910239:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24301568->24305663] (req 24301568->24305663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.910247:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24301568->24305663] (req 24301568->24305663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.910254:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760776.910258:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24301568->24305663] (req 24301568->24305663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.910267:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24301568->24305663] (req 24301568->24305663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.910287:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24301568->24305663] (req 24301568->24305663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.910295:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf3826396209bf8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24301568->24305663] (req 24301568->24305663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760776.910425:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484328704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37566us (37803us total) trans 4294973251 rc 0/0 00000100:00100000:0.0:1789760776.910436:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5962 00000100:00100000:0.0:1789760776.910445:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766159100385 time 766203773809 state 0x2 ref 0 00000100:00100000:2.0:1789760776.913888:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760776.914016:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484328832 00000100:00100000:3.0:1789760776.914027:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484328832 00000100:00100000:3.0:1789760776.914111:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766159100385 time 766203773809 state 0x0 ref 1 00000100:00100000:3.0:1789760776.914127:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766204034225 time 766203773809 state 0x1 ref 1 00000100:00100000:3.0:1789760776.914155:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766204034225 time 766245085820 state 0x0 ref 1 00000100:00100000:3.0:1789760776.914168:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484328832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760776.914218:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24305664, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760776.914227:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.914242:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:3.0:1789760776.914247:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760776.914254:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24305664->24309759] (req 24305664->24309759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.914269:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24305664->24309759] (req 24305664->24309759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760776.943618:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973252 is committed 00002000:00010000:3.0:1789760776.943912:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24305664 -> 24309760 00010000:00010000:3.0:1789760776.943927:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24305664->24309759] (req 24305664->24309759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.943940:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24305664->24309759] (req 24305664->24309759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.943948:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24305664->24309759] (req 24305664->24309759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.943956:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760776.943960:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24305664->24309759] (req 24305664->24309759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.943970:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24305664->24309759] (req 24305664->24309759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.943989:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24305664->24309759] (req 24305664->24309759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760776.943996:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209bff lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24305664->24309759] (req 24305664->24309759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760776.944130:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484328832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29959us (30248us total) trans 4294973252 rc 0/0 00000100:00100000:3.0:1789760776.944144:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5963 00000100:00100000:3.0:1789760776.944153:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766204034225 time 766245085820 state 0x2 ref 0 00000100:00100000:2.0:1789760776.950468:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760776.950556:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484328960 00000100:00100000:0.0:1789760776.950572:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484328960 00000100:00100000:0.0:1789760776.950656:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766204034225 time 766245085820 state 0x0 ref 1 00000100:00100000:0.0:1789760776.950671:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766245346236 time 766245085820 state 0x1 ref 1 00000100:00100000:0.0:1789760776.950707:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766245346236 time 766281638347 state 0x0 ref 1 00000100:00100000:0.0:1789760776.950722:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484328960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760776.950774:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24309760, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:0.0:1789760776.950782:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.950798:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:0.0:1789760776.950802:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760776.950809:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24309760->24313855] (req 24309760->24313855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.950821:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24309760->24313855] (req 24309760->24313855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760776.981795:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973253 is committed 00002000:00010000:0.0:1789760776.982161:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24309760 -> 24313856 00010000:00010000:0.0:1789760776.982177:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24309760->24313855] (req 24309760->24313855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.982190:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24309760->24313855] (req 24309760->24313855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.982206:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24309760->24313855] (req 24309760->24313855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.982214:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760776.982219:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24309760->24313855] (req 24309760->24313855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.982228:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24309760->24313855] (req 24309760->24313855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.982247:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24309760->24313855] (req 24309760->24313855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.982255:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c06 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24309760->24313855] (req 24309760->24313855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760776.982382:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484328960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31658us (31923us total) trans 4294973253 rc 0/0 00000100:00100000:0.0:1789760776.982392:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5964 00000100:00100000:0.0:1789760776.982400:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766245346236 time 766281638347 state 0x2 ref 0 00000100:00100000:2.0:1789760776.989680:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760776.990660:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484329088 00000100:00100000:0.0:1789760776.990673:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484329088 00000100:00100000:0.0:1789760776.990738:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766245346236 time 766281638347 state 0x0 ref 1 00000100:00100000:0.0:1789760776.990753:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766281898763 time 766281638347 state 0x1 ref 1 00000100:00100000:0.0:1789760776.990790:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766281898763 time 766321719878 state 0x0 ref 1 00000100:00100000:0.0:1789760776.990803:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484329088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760776.990851:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24313856, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:0.0:1789760776.990860:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.990877:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:0.0:1789760776.990881:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760776.990888:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24313856->24317951] (req 24313856->24317951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760776.990900:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24313856->24317951] (req 24313856->24317951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760777.034792:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24313856 -> 24317952 00010000:00010000:2.0:1789760777.034811:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24313856->24317951] (req 24313856->24317951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.034825:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24313856->24317951] (req 24313856->24317951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.034852:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24313856->24317951] (req 24313856->24317951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.034860:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760777.034865:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24313856->24317951] (req 24313856->24317951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.034873:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24313856->24317951] (req 24313856->24317951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.034893:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24313856->24317951] (req 24313856->24317951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.034901:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c0d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24313856->24317951] (req 24313856->24317951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760777.034991:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484329088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44186us (45316us total) trans 4294973254 rc 0/0 00000100:00100000:2.0:1789760777.035002:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5965 00000100:00100000:2.0:1789760777.035011:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766281898763 time 766321719878 state 0x2 ref 0 00000001:00080000:2.0:1789760777.035103:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973254 is committed 00000100:00100000:2.0:1789760777.041370:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760777.042744:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484329216 00000100:00100000:1.0:1789760777.042756:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484329216 00000100:00100000:1.0:1789760777.042817:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766281898763 time 766321719878 state 0x0 ref 1 00000100:00100000:1.0:1789760777.042833:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766321980294 time 766321719878 state 0x1 ref 1 00000100:00100000:1.0:1789760777.042870:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766321980294 time 766373800797 state 0x0 ref 1 00000100:00100000:1.0:1789760777.042884:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484329216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760777.042935:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24317952, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:1.0:1789760777.042944:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.042959:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:1.0:1789760777.042964:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760777.042970:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24317952->24322047] (req 24317952->24322047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.042984:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24317952->24322047] (req 24317952->24322047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760777.073495:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973255 is committed 00002000:00010000:2.0:1789760777.073672:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24317952 -> 24322048 00010000:00010000:2.0:1789760777.073689:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24317952->24322047] (req 24317952->24322047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.073702:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24317952->24322047] (req 24317952->24322047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.073711:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24317952->24322047] (req 24317952->24322047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.073718:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760777.073723:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24317952->24322047] (req 24317952->24322047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.073732:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24317952->24322047] (req 24317952->24322047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.073751:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24317952->24322047] (req 24317952->24322047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.073759:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf3826396209c14 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24317952->24322047] (req 24317952->24322047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760777.073869:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484329216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30983us (32505us total) trans 4294973255 rc 0/0 00000100:00100000:2.0:1789760777.073880:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5966 00000100:00100000:2.0:1789760777.073890:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766321980294 time 766373800797 state 0x2 ref 0 00000100:00100000:1.0:1789760777.083160:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760777.083329:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484329344 00000100:00100000:3.0:1789760777.083346:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484329344 00000100:00100000:3.0:1789760777.083421:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766321980294 time 766373800797 state 0x0 ref 1 00000100:00100000:3.0:1789760777.083440:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766374061213 time 766373800797 state 0x1 ref 1 00000100:00100000:3.0:1789760777.083474:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766374061213 time 766414405304 state 0x0 ref 1 00000100:00100000:3.0:1789760777.083493:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484329344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760777.083559:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24322048, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:3.0:1789760777.083569:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.083590:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:3.0:1789760777.083595:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760777.083602:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24322048->24326143] (req 24322048->24326143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.083618:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24322048->24326143] (req 24322048->24326143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.100260:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760777.113382:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973256 is committed 00002000:00010000:0.0:1789760777.113471:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24322048 -> 24326144 00010000:00010000:0.0:1789760777.113487:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24322048->24326143] (req 24322048->24326143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.113500:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24322048->24326143] (req 24322048->24326143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.113510:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24322048->24326143] (req 24322048->24326143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.113522:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760777.113527:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24322048->24326143] (req 24322048->24326143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.113536:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24322048->24326143] (req 24322048->24326143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.113555:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24322048->24326143] (req 24322048->24326143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.113562:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c1b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24322048->24326143] (req 24322048->24326143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760777.113644:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484329344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30153us (30488us total) trans 4294973256 rc 0/0 00000100:00100000:0.0:1789760777.113653:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5967 00000100:00100000:0.0:1789760777.113662:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766374061213 time 766414405304 state 0x2 ref 0 00000100:00100000:3.0:1789760777.120272:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760777.120428:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484329472 00000100:00100000:0.0:1789760777.120438:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484329472 00000100:00100000:0.0:1789760777.120498:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766374061213 time 766414405304 state 0x0 ref 1 00000100:00100000:0.0:1789760777.120513:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766414665720 time 766414405304 state 0x1 ref 1 00000100:00100000:0.0:1789760777.120541:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766414665720 time 766451472577 state 0x0 ref 1 00000100:00100000:0.0:1789760777.120555:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484329472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760777.120603:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24326144, mtime: 0x6aad9508, atime: 0x0, ctime: 0x6aad9508, blocks: 0xc007 00010000:00010000:0.0:1789760777.120612:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.120627:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:0.0:1789760777.120631:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760777.120637:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24326144->24330239] (req 24326144->24330239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.120649:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24326144->24330239] (req 24326144->24330239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760777.164062:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973257 is committed 00002000:00010000:1.0:1789760777.164354:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24326144 -> 24330240 00002000:00010000:1.0:1789760777.164360:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760776 -> 1789760777 00002000:00010000:1.0:1789760777.164363:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760776 -> 1789760777 00010000:00010000:1.0:1789760777.164375:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24326144->24330239] (req 24326144->24330239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.164388:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24326144->24330239] (req 24326144->24330239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.164396:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24326144->24330239] (req 24326144->24330239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.164405:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760777.164591:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24326144->24330239] (req 24326144->24330239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.164602:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24326144->24330239] (req 24326144->24330239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.164621:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24326144->24330239] (req 24326144->24330239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.164629:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209c22 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24326144->24330239] (req 24326144->24330239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760777.164728:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484329472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44170us (44470us total) trans 4294973257 rc 0/0 00000100:00100000:1.0:1789760777.164737:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5968 00000100:00100000:1.0:1789760777.164747:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766414665720 time 766451472577 state 0x2 ref 0 00000100:00100000:2.0:1789760777.168310:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760777.168377:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484329600 00000100:00100000:0.0:1789760777.168385:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484329600 00000100:00100000:0.0:1789760777.168442:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766414665720 time 766451472577 state 0x0 ref 1 00000100:00100000:0.0:1789760777.168453:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766451732993 time 766451472577 state 0x1 ref 1 00000100:00100000:0.0:1789760777.168502:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766451732993 time 766499433658 state 0x0 ref 1 00000100:00100000:0.0:1789760777.168511:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484329600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760777.168551:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24330240, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:0.0:1789760777.168558:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.168574:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:0.0:1789760777.168577:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760777.168581:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24330240->24334335] (req 24330240->24334335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.168590:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24330240->24334335] (req 24330240->24334335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760777.193799:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24330240 -> 24334336 00010000:00010000:1.0:1789760777.193818:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24330240->24334335] (req 24330240->24334335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.193841:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24330240->24334335] (req 24330240->24334335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.193855:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24330240->24334335] (req 24330240->24334335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.193862:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760777.193867:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24330240->24334335] (req 24330240->24334335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.193875:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24330240->24334335] (req 24330240->24334335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.193896:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24330240->24334335] (req 24330240->24334335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.193903:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c29 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24330240->24334335] (req 24330240->24334335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760777.194007:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484329600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25492us (25699us total) trans 4294973258 rc 0/0 00000100:00100000:1.0:1789760777.194017:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5969 00000100:00100000:1.0:1789760777.194027:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766451732993 time 766499433658 state 0x2 ref 0 00000001:00080000:3.0:1789760777.194634:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973258 is committed 00000100:00100000:2.0:1789760777.198043:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760777.199776:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484329728 00000100:00100000:1.0:1789760777.199788:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484329728 00000100:00100000:1.0:1789760777.199916:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766451732993 time 766499433658 state 0x0 ref 1 00000100:00100000:1.0:1789760777.199930:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766499694074 time 766499433658 state 0x1 ref 1 00000100:00100000:1.0:1789760777.199971:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766499694074 time 766530902214 state 0x0 ref 1 00000100:00100000:1.0:1789760777.199984:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484329728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760777.200032:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24334336, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:1.0:1789760777.200040:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.200087:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:1.0:1789760777.200091:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760777.200098:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24334336->24338431] (req 24334336->24338431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.200113:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24334336->24338431] (req 24334336->24338431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760777.233385:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973259 is committed 00002000:00010000:0.0:1789760777.233491:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24334336 -> 24338432 00010000:00010000:0.0:1789760777.233506:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24334336->24338431] (req 24334336->24338431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.233520:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24334336->24338431] (req 24334336->24338431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.233529:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24334336->24338431] (req 24334336->24338431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.233537:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760777.233542:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24334336->24338431] (req 24334336->24338431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.233551:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24334336->24338431] (req 24334336->24338431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.233583:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24334336->24338431] (req 24334336->24338431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.233592:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209c30 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24334336->24338431] (req 24334336->24338431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760777.233699:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484329728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33712us (35661us total) trans 4294973259 rc 0/0 00000100:00100000:0.0:1789760777.233709:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5970 00000100:00100000:0.0:1789760777.233719:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766499694074 time 766530902214 state 0x2 ref 0 00000100:00100000:0.0:1789760777.237910:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760777.238106:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484329856 00000100:00100000:3.0:1789760777.238116:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484329856 00000100:00100000:3.0:1789760777.238166:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766499694074 time 766530902214 state 0x0 ref 1 00000100:00100000:3.0:1789760777.238179:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766531162630 time 766530902214 state 0x1 ref 1 00000100:00100000:3.0:1789760777.238208:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766531162630 time 766569139855 state 0x0 ref 1 00000100:00100000:3.0:1789760777.238220:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484329856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760777.238264:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24338432, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:3.0:1789760777.238272:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.238286:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:3.0:1789760777.238290:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760777.238295:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24338432->24342527] (req 24338432->24342527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.238308:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24338432->24342527] (req 24338432->24342527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760777.265325:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973260 is committed 00002000:00010000:3.0:1789760777.265388:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24338432 -> 24342528 00010000:00010000:3.0:1789760777.265403:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24338432->24342527] (req 24338432->24342527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.265422:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24338432->24342527] (req 24338432->24342527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.265432:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24338432->24342527] (req 24338432->24342527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.265439:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760777.265443:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24338432->24342527] (req 24338432->24342527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.265452:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24338432->24342527] (req 24338432->24342527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.265471:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24338432->24342527] (req 24338432->24342527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.265480:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c37 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24338432->24342527] (req 24338432->24342527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760777.265571:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484329856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27348us (27665us total) trans 4294973260 rc 0/0 00000100:00100000:3.0:1789760777.265581:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5971 00000100:00100000:3.0:1789760777.265591:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766531162630 time 766569139855 state 0x2 ref 0 00000100:00100000:2.0:1789760777.275487:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760777.275594:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484329984 00000100:00100000:0.0:1789760777.275604:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484329984 00000100:00100000:0.0:1789760777.275667:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766531162630 time 766569139855 state 0x0 ref 1 00000100:00100000:0.0:1789760777.276042:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766569400271 time 766569139855 state 0x1 ref 1 00000100:00100000:0.0:1789760777.276132:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766569400271 time 766607061759 state 0x0 ref 1 00000100:00100000:0.0:1789760777.276319:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484329984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760777.276378:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24342528, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:0.0:1789760777.276387:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.276403:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:0.0:1789760777.276407:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760777.276413:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24342528->24346623] (req 24342528->24346623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.276428:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24342528->24346623] (req 24342528->24346623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760777.313946:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24342528 -> 24346624 00010000:00010000:2.0:1789760777.313973:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24342528->24346623] (req 24342528->24346623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.313987:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24342528->24346623] (req 24342528->24346623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.313996:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24342528->24346623] (req 24342528->24346623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.314003:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760777.314008:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24342528->24346623] (req 24342528->24346623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.314020:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24342528->24346623] (req 24342528->24346623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.314039:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24342528->24346623] (req 24342528->24346623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.314058:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf3826396209c3e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24342528->24346623] (req 24342528->24346623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760777.314094:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973261 is committed 00000100:00100000:2.0:1789760777.314266:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484329984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37948us (38783us total) trans 4294973261 rc 0/0 00000100:00100000:2.0:1789760777.314277:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5972 00000100:00100000:2.0:1789760777.314286:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766569400271 time 766607061759 state 0x2 ref 0 00000100:00100000:2.0:1789760777.318527:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760777.318652:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484330112 00000100:00100000:1.0:1789760777.318663:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484330112 00000100:00100000:1.0:1789760777.318725:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766569400271 time 766607061759 state 0x0 ref 1 00000100:00100000:1.0:1789760777.318740:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766607322175 time 766607061759 state 0x1 ref 1 00000100:00100000:1.0:1789760777.318770:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766607322175 time 766649700680 state 0x0 ref 1 00000100:00100000:1.0:1789760777.318783:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484330112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760777.318832:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24346624, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:1.0:1789760777.318842:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.318857:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:1.0:1789760777.318861:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760777.318868:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24346624->24350719] (req 24346624->24350719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.318881:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24346624->24350719] (req 24346624->24350719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760777.365421:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24346624 -> 24350720 00010000:00010000:2.0:1789760777.365441:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24346624->24350719] (req 24346624->24350719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.365452:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24346624->24350719] (req 24346624->24350719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.365460:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24346624->24350719] (req 24346624->24350719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.365467:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760777.365473:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24346624->24350719] (req 24346624->24350719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.365482:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24346624->24350719] (req 24346624->24350719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.365572:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24346624->24350719] (req 24346624->24350719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.365585:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209c45 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24346624->24350719] (req 24346624->24350719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760777.365809:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973262 is committed 00000100:00100000:2.0:1789760777.366022:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484330112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47236us (47499us total) trans 4294973262 rc 0/0 00000100:00100000:2.0:1789760777.366033:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5973 00000100:00100000:2.0:1789760777.366043:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766607322175 time 766649700680 state 0x2 ref 0 00000100:00100000:1.0:1789760777.373554:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760777.373998:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484330240 00000100:00100000:0.0:1789760777.374008:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484330240 00000100:00100000:0.0:1789760777.374199:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766607322175 time 766649700680 state 0x0 ref 1 00000100:00100000:0.0:1789760777.374216:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766649961096 time 766649700680 state 0x1 ref 1 00000100:00100000:0.0:1789760777.374298:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766649961096 time 766705227800 state 0x0 ref 1 00000100:00100000:0.0:1789760777.374313:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484330240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760777.374374:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24350720, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:0.0:1789760777.374382:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.374397:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:0.0:1789760777.374401:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760777.374406:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24350720->24354815] (req 24350720->24354815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.374420:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24350720->24354815] (req 24350720->24354815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760777.419313:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24350720 -> 24354816 00010000:00010000:2.0:1789760777.419333:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24350720->24354815] (req 24350720->24354815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.419347:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24350720->24354815] (req 24350720->24354815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.419356:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24350720->24354815] (req 24350720->24354815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.419363:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760777.419368:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24350720->24354815] (req 24350720->24354815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.419377:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24350720->24354815] (req 24350720->24354815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.419403:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24350720->24354815] (req 24350720->24354815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.419411:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209c4c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24350720->24354815] (req 24350720->24354815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760777.419519:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484330240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45204us (45970us total) trans 4294973263 rc 0/0 00000100:00100000:2.0:1789760777.419530:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5974 00000100:00100000:2.0:1789760777.419539:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766649961096 time 766705227800 state 0x2 ref 0 00000001:00080000:3.0:1789760777.419668:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973263 is committed 00000100:00100000:1.0:1789760777.424171:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760777.424225:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484330368 00000100:00100000:3.0:1789760777.424236:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484330368 00000100:00100000:3.0:1789760777.424302:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766649961096 time 766705227800 state 0x0 ref 1 00000100:00100000:3.0:1789760777.424317:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766705488216 time 766705227800 state 0x1 ref 1 00000100:00100000:3.0:1789760777.424505:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766705488216 time 766755435584 state 0x0 ref 1 00000100:00100000:3.0:1789760777.424519:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484330368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760777.424569:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24354816, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:3.0:1789760777.424578:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.424594:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:3.0:1789760777.424599:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760777.424604:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24354816->24358911] (req 24354816->24358911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.424618:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24354816->24358911] (req 24354816->24358911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760777.457388:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24354816 -> 24358912 00010000:00010000:3.0:1789760777.457408:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24354816->24358911] (req 24354816->24358911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.457428:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24354816->24358911] (req 24354816->24358911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.457437:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24354816->24358911] (req 24354816->24358911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.457444:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760777.457449:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24354816->24358911] (req 24354816->24358911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.457458:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24354816->24358911] (req 24354816->24358911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.457476:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24354816->24358911] (req 24354816->24358911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.457484:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209c53 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24354816->24358911] (req 24354816->24358911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760777.457611:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484330368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33089us (33439us total) trans 4294973264 rc 0/0 00000100:00100000:3.0:1789760777.457622:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5975 00000100:00100000:3.0:1789760777.457631:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766705488216 time 766755435584 state 0x2 ref 0 00000001:00080000:3.0:1789760777.457670:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973264 is committed 00000100:00100000:0.0:1789760777.467223:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760777.467457:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484330496 00000100:00100000:2.0:1789760777.467467:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484330496 00000100:00100000:2.0:1789760777.467788:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766705488216 time 766755435584 state 0x0 ref 1 00000100:00100000:2.0:1789760777.467810:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766755696000 time 766755435584 state 0x1 ref 1 00000100:00100000:2.0:1789760777.467880:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766755696000 time 766798810412 state 0x0 ref 1 00000100:00100000:2.0:1789760777.467895:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484330496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760777.467950:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24358912, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:2.0:1789760777.467959:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.467975:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:2.0:1789760777.467980:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760777.467986:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24358912->24363007] (req 24358912->24363007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.468000:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24358912->24363007] (req 24358912->24363007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760777.502379:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973265 is committed 00002000:00010000:2.0:1789760777.502589:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24358912 -> 24363008 00010000:00010000:2.0:1789760777.502604:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24358912->24363007] (req 24358912->24363007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.502616:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24358912->24363007] (req 24358912->24363007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.502624:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24358912->24363007] (req 24358912->24363007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.502631:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760777.502635:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24358912->24363007] (req 24358912->24363007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.502645:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24358912->24363007] (req 24358912->24363007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.502665:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24358912->24363007] (req 24358912->24363007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.502673:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c5a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24358912->24363007] (req 24358912->24363007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760777.502794:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484330496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34897us (35577us total) trans 4294973265 rc 0/0 00000100:00100000:2.0:1789760777.502803:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5976 00000100:00100000:2.0:1789760777.502813:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766755696000 time 766798810412 state 0x2 ref 0 00000100:00100000:1.0:1789760777.515760:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760777.515950:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484330624 00000100:00100000:2.0:1789760777.515960:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484330624 00000100:00100000:2.0:1789760777.516012:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766755696000 time 766798810412 state 0x0 ref 1 00000100:00100000:2.0:1789760777.516027:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766799070828 time 766798810412 state 0x1 ref 1 00000100:00100000:2.0:1789760777.516192:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766799070828 time 766846997484 state 0x0 ref 1 00000100:00100000:2.0:1789760777.516207:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484330624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760777.516256:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24363008, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:2.0:1789760777.516265:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.516294:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:2.0:1789760777.516298:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760777.516304:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24363008->24367103] (req 24363008->24367103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.516317:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24363008->24367103] (req 24363008->24367103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760777.565317:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973266 is committed 00002000:00010000:2.0:1789760777.565470:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24363008 -> 24367104 00010000:00010000:2.0:1789760777.565485:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24363008->24367103] (req 24363008->24367103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.565498:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24363008->24367103] (req 24363008->24367103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.565507:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24363008->24367103] (req 24363008->24367103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.565515:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760777.565520:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24363008->24367103] (req 24363008->24367103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.565530:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24363008->24367103] (req 24363008->24367103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.565550:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24363008->24367103] (req 24363008->24367103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.565558:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209c61 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24363008->24367103] (req 24363008->24367103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760777.565646:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484330624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49437us (49892us total) trans 4294973266 rc 0/0 00000100:00100000:2.0:1789760777.565660:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5977 00000100:00100000:2.0:1789760777.565668:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766799070828 time 766846997484 state 0x2 ref 0 00000100:00100000:2.0:1789760777.568732:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760777.568813:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484330752 00000100:00100000:0.0:1789760777.568822:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484330752 00000100:00100000:0.0:1789760777.568933:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766799070828 time 766846997484 state 0x0 ref 1 00000100:00100000:0.0:1789760777.568947:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766847257900 time 766846997484 state 0x1 ref 1 00000100:00100000:0.0:1789760777.568997:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766847257900 time 766899928026 state 0x0 ref 1 00000100:00100000:0.0:1789760777.569009:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484330752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760777.569078:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24367104, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:0.0:1789760777.569087:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.569102:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:0.0:1789760777.569106:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760777.569112:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24367104->24371199] (req 24367104->24371199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.569126:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24367104->24371199] (req 24367104->24371199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760777.604651:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973267 is committed 00002000:00010000:1.0:1789760777.604763:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24367104 -> 24371200 00010000:00010000:1.0:1789760777.604778:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24367104->24371199] (req 24367104->24371199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.604790:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24367104->24371199] (req 24367104->24371199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.604798:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24367104->24371199] (req 24367104->24371199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.604806:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760777.604811:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24367104->24371199] (req 24367104->24371199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.604821:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24367104->24371199] (req 24367104->24371199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.604906:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24367104->24371199] (req 24367104->24371199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.604914:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209c68 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24367104->24371199] (req 24367104->24371199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760777.605013:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484330752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36001us (36284us total) trans 4294973267 rc 0/0 00000100:00100000:1.0:1789760777.605024:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5978 00000100:00100000:1.0:1789760777.605034:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766847257900 time 766899928026 state 0x2 ref 0 00000100:00100000:0.0:1789760777.609460:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760777.609532:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484330880 00000100:00100000:2.0:1789760777.609542:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484330880 00000100:00100000:2.0:1789760777.609595:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766847257900 time 766899928026 state 0x0 ref 1 00000100:00100000:2.0:1789760777.609611:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766900188442 time 766899928026 state 0x1 ref 1 00000100:00100000:2.0:1789760777.609643:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766900188442 time 766940573824 state 0x0 ref 1 00000100:00100000:2.0:1789760777.609655:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484330880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760777.609703:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24371200, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:2.0:1789760777.609712:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.609726:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:2.0:1789760777.609731:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760777.609737:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24371200->24375295] (req 24371200->24375295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.609750:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24371200->24375295] (req 24371200->24375295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760777.612428:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfbb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508936704:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.612489:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760777.612535:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a32d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508937216:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.612555:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760777.612576:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb724000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508937728:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.612593:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760777.612617:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [3->2] 00000100:00100000:2.0:1789760777.612622:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a31880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508936832:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.612645:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760777.612661:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508937344:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.612680:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760777.612696:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5082a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508937856:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.612714:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760777.612743:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508936704 00000100:00100000:2.0:1789760777.612750:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508936704 00000100:00100000:2.0:1789760777.612784:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508937216 00000100:00100000:2.0:1789760777.612787:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508937216 00000100:00100000:2.0:1789760777.612809:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508937728 00000100:00100000:2.0:1789760777.612811:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508937728 00000100:00100000:2.0:1789760777.612823:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508937344 00000100:00100000:2.0:1789760777.612825:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508937344 00000100:00100000:2.0:1789760777.612849:0:14751:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508937856 00000100:00100000:2.0:1789760777.612852:0:14751:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508937856 00000100:00100000:2.0:1789760777.612866:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 985 00000100:00100000:2.0:1789760777.612874:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12af100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508936704:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.612925:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12af100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508936704:12345-0@lo:400:kworker.0 Request processed in 51us (435us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760777.612932:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 985 00000100:00100000:2.0:1789760777.612949:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 986 00000100:00100000:2.0:1789760777.612955:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12aca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508937216:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.612982:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12aca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508937216:12345-0@lo:400:kworker.0 Request processed in 27us (426us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760777.612988:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 986 00000100:00100000:2.0:1789760777.612999:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 987 00000100:00100000:2.0:1789760777.613005:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12af800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508937728:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.613027:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12af800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508937728:12345-0@lo:400:kworker.0 Request processed in 23us (433us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760777.613033:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 987 00000100:00100000:2.0:1789760777.613042:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 988 00000100:00100000:2.0:1789760777.613059:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ac380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508937344:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.613102:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ac380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508937344:12345-0@lo:400:kworker.0 Request processed in 44us (421us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760777.613108:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 988 00000100:00100000:2.0:1789760777.613120:0:14751:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 989 00000100:00100000:2.0:1789760777.613126:0:14751:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12adf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508937856:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.613151:0:14751:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12adf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508937856:12345-0@lo:400:kworker.0 Request processed in 25us (436us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760777.613157:0:14751:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 989 00000100:00100000:2.0:1789760777.613178:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508936832 00000100:00100000:2.0:1789760777.613182:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508936832 00000100:00100000:2.0:1789760777.613208:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 467 00000100:00100000:2.0:1789760777.613214:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ad180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508936832:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.613242:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ad180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508936832:12345-0@lo:400:kworker.0 Request processed in 29us (596us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760777.613248:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 467 00000100:00080000:2.0:1789760777.613269:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760777.613272:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:2.0:1789760777.613302:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a31880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508936832:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.613313:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a30e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508937344:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.613322:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5082a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508937856:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.613331:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfbb7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508936704:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.613339:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a32d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508937216:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.613348:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb724000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508937728:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613439:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfbb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3479:1876699508936576:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613472:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760777.613503:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a31f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508937088:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613523:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760777.613538:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb725180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508937600:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613557:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760777.613573:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c12ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508938112:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613592:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760777.613608:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1789760777.613614:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508936960:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613631:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760777.613643:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4a32680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508937472:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613659:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760777.613672:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5083b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508937984:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613688:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760777.613714:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508936960 00000100:00100000:3.0:1789760777.613718:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508936960 00000100:00100000:3.0:1789760777.613741:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 990 00000100:00100000:3.0:1789760777.613748:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b18e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508936960:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613780:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508936960:12345-0@lo:400:kworker.0 Request processed in 32us (147us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760777.613786:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 990 00000100:00100000:3.0:1789760777.613804:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508937088 00000100:00100000:3.0:1789760777.613807:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508937088 00000100:00100000:3.0:1789760777.613824:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 468 00000100:00100000:3.0:1789760777.613830:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508937088:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613859:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508937088:12345-0@lo:400:kworker.0 Request processed in 29us (335us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760777.613865:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 468 00000100:00100000:3.0:1789760777.613880:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508937600 00000100:00100000:3.0:1789760777.613883:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508937600 00000100:00100000:3.0:1789760777.613909:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508938112 00000100:00100000:3.0:1789760777.613911:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508938112 00000100:00100000:3.0:1789760777.613922:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508937472 00000100:00100000:3.0:1789760777.613925:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508937472 00000100:00100000:3.0:1789760777.613935:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508937984 00000100:00100000:3.0:1789760777.613937:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508937984 00000100:00100000:3.0:1789760777.613949:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 343 00000100:00100000:3.0:1789760777.613954:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508937600:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.613983:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508937600:12345-0@lo:400:kworker.0 Request processed in 28us (424us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760777.613989:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 343 00000100:00100000:3.0:1789760777.613999:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 344 00000100:00100000:3.0:1789760777.614004:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508938112:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.614027:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508938112:12345-0@lo:400:kworker.0 Request processed in 23us (435us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760777.614033:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 344 00000100:00100000:3.0:1789760777.614059:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 345 00000100:00100000:3.0:1789760777.614090:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b18380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508937472:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.614116:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508937472:12345-0@lo:400:kworker.0 Request processed in 51us (455us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760777.614122:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 345 00000100:00100000:3.0:1789760777.614134:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 346 00000100:00100000:3.0:1789760777.614141:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508937984:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.614167:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508937984:12345-0@lo:400:kworker.0 Request processed in 27us (478us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760777.614173:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 346 00000100:00100000:3.0:1789760777.614326:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a30700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508936960:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.614337:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a32680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508937472:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.614345:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5083b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508937984:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.614352:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4a31f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508937088:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.614360:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb725180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508937600:0@lo:400:kworker.0 00000100:00100000:3.0:1789760777.614368:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c12ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508938112:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.614845:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508936576 00000100:00100000:2.0:1789760777.614850:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508936576 00000100:00100000:2.0:1789760777.614885:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 492 00000100:00100000:2.0:1789760777.614893:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8039850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3479:x1876699508936576:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.614936:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8039850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3479:x1876699508936576:12345-0@lo:400:kworker.0 Request processed in 45us (1463us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760777.614943:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 492 00000100:00080000:2.0:1789760777.614971:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:2.0:1789760777.614974:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760777.614995:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfbb4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3479:1876699508936576:0@lo:400:kworker.0 00000100:00100000:2.0:1789760777.617038:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:2.0:1789760777.617079:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508938240:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760777.617140:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760777.617229:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508938240 00000100:00100000:3.0:1789760777.617237:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508938240 00000100:00100000:3.0:1789760777.617285:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 408 00000100:00100000:3.0:1789760777.617293:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c12ad880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508938240:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:3.0:1789760777.617327:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:3.0:1789760777.617371:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c12ad880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508938240:12345-0@lo:13:osp-pre-1-0.0 Request processed in 78us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760777.617378:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 408 00000100:00100000:2.0:1789760777.617464:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508938240:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760777.619240:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f4f10700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508938368:0@lo:13:osp-pre-1-1.0 00000100:00100000:1.0:1789760777.619299:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760777.619479:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508938368 00000100:00100000:2.0:1789760777.619487:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508938368 00000100:00100000:2.0:1789760777.619525:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 409 00000100:00100000:2.0:1789760777.619533:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508938368:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:2.0:1789760777.619552:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760777.619591:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508938368:12345-0@lo:13:osp-pre-1-1.0 Request processed in 59us (291us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760777.619598:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 409 00000100:00100000:0.0:1789760777.621396:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f4f10700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508938368:0@lo:13:osp-pre-1-1.0 00000001:00080000:2.0:1789760777.640546:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973268 is committed 00002000:00010000:2.0:1789760777.640608:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24371200 -> 24375296 00010000:00010000:2.0:1789760777.640622:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24371200->24375295] (req 24371200->24375295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.640634:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24371200->24375295] (req 24371200->24375295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.640643:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24371200->24375295] (req 24371200->24375295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.640650:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760777.640655:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24371200->24375295] (req 24371200->24375295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.640665:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24371200->24375295] (req 24371200->24375295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.640686:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24371200->24375295] (req 24371200->24375295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.640694:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c6f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24371200->24375295] (req 24371200->24375295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760777.640775:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484330880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31117us (31320us total) trans 4294973268 rc 0/0 00000100:00100000:2.0:1789760777.640785:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5979 00000100:00100000:2.0:1789760777.640795:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766900188442 time 766940573824 state 0x2 ref 0 00000100:00100000:2.0:1789760777.643446:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760777.643529:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484331008 00000100:00100000:0.0:1789760777.643539:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484331008 00000100:00100000:0.0:1789760777.643598:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766900188442 time 766940573824 state 0x0 ref 1 00000100:00100000:0.0:1789760777.643616:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766940834240 time 766940573824 state 0x1 ref 1 00000100:00100000:0.0:1789760777.643645:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766940834240 time 766974575925 state 0x0 ref 1 00000100:00100000:0.0:1789760777.643658:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484331008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760777.643705:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24375296, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:0.0:1789760777.643713:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.643727:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=766s v=(0 0) 00010000:00010000:0.0:1789760777.643731:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760777.643737:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24375296->24379391] (req 24375296->24379391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.643748:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24375296->24379391] (req 24375296->24379391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760777.669292:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973269 is committed 00002000:00010000:3.0:1789760777.669386:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24375296 -> 24379392 00010000:00010000:3.0:1789760777.669402:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24375296->24379391] (req 24375296->24379391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.669414:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24375296->24379391] (req 24375296->24379391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.669423:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24375296->24379391] (req 24375296->24379391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.669430:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760777.669435:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24375296->24379391] (req 24375296->24379391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.669445:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24375296->24379391] (req 24375296->24379391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.669468:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24375296->24379391] (req 24375296->24379391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.669475:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209c76 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24375296->24379391] (req 24375296->24379391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760777.669604:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484331008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25944us (26163us total) trans 4294973269 rc 0/0 00000100:00100000:3.0:1789760777.669614:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5980 00000100:00100000:3.0:1789760777.669624:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766940834240 time 766974575925 state 0x2 ref 0 00000100:00100000:2.0:1789760777.673015:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760777.673642:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484331136 00000100:00100000:1.0:1789760777.673653:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484331136 00000100:00100000:2.0:1789760777.673776:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766940834240 time 766974575925 state 0x0 ref 1 00000100:00100000:2.0:1789760777.673791:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766974836341 time 766974575925 state 0x1 ref 1 00000100:00100000:2.0:1789760777.673909:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766974836341 time 767004839518 state 0x0 ref 1 00000100:00100000:2.0:1789760777.673924:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484331136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760777.673970:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24379392, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:2.0:1789760777.673980:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.673996:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:2.0:1789760777.674000:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760777.674007:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24379392->24383487] (req 24379392->24383487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.674022:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24379392->24383487] (req 24379392->24383487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760777.705349:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973270 is committed 00002000:00010000:3.0:1789760777.705483:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24379392 -> 24383488 00010000:00010000:3.0:1789760777.705497:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24379392->24383487] (req 24379392->24383487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.705515:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24379392->24383487] (req 24379392->24383487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.705524:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24379392->24383487] (req 24379392->24383487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.705532:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760777.705537:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24379392->24383487] (req 24379392->24383487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.705547:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24379392->24383487] (req 24379392->24383487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.705565:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24379392->24383487] (req 24379392->24383487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.705573:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209c7d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24379392->24383487] (req 24379392->24383487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760777.705656:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484331136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31730us (32646us total) trans 4294973270 rc 0/0 00000100:00100000:3.0:1789760777.705665:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5981 00000100:00100000:3.0:1789760777.705675:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766974836341 time 767004839518 state 0x2 ref 0 00000100:00100000:2.0:1789760777.708456:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760777.708527:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484331264 00000100:00100000:1.0:1789760777.708537:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484331264 00000100:00100000:1.0:1789760777.708591:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 766974836341 time 767004839518 state 0x0 ref 1 00000100:00100000:1.0:1789760777.708606:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767005099934 time 767004839518 state 0x1 ref 1 00000100:00100000:1.0:1789760777.708670:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767005099934 time 767039600327 state 0x0 ref 1 00000100:00100000:1.0:1789760777.708684:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484331264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760777.708732:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24383488, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:1.0:1789760777.708741:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.708756:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:1.0:1789760777.708760:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760777.708767:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24383488->24387583] (req 24383488->24387583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.708781:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24383488->24387583] (req 24383488->24387583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760777.735426:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973271 is committed 00002000:00010000:1.0:1789760777.735489:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24383488 -> 24387584 00010000:00010000:1.0:1789760777.735503:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24383488->24387583] (req 24383488->24387583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.735516:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24383488->24387583] (req 24383488->24387583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.735525:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24383488->24387583] (req 24383488->24387583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.735532:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760777.735537:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24383488->24387583] (req 24383488->24387583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.735546:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24383488->24387583] (req 24383488->24387583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.735565:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24383488->24387583] (req 24383488->24387583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.735571:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209c84 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24383488->24387583] (req 24383488->24387583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760777.735776:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484331264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27089us (27324us total) trans 4294973271 rc 0/0 00000100:00100000:1.0:1789760777.735787:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5982 00000100:00100000:1.0:1789760777.735797:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767005099934 time 767039600327 state 0x2 ref 0 00000100:00100000:2.0:1789760777.741181:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760777.741280:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484331392 00000100:00100000:0.0:1789760777.741290:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484331392 00000100:00100000:0.0:1789760777.741357:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767005099934 time 767039600327 state 0x0 ref 1 00000100:00100000:0.0:1789760777.741372:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767039860743 time 767039600327 state 0x1 ref 1 00000100:00100000:0.0:1789760777.741401:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767039860743 time 767072331803 state 0x0 ref 1 00000100:00100000:0.0:1789760777.741418:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484331392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760777.741464:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24387584, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:0.0:1789760777.741473:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.741490:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:0.0:1789760777.741494:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760777.741500:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24387584->24391679] (req 24387584->24391679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.741513:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24387584->24391679] (req 24387584->24391679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760777.774735:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973272 is committed 00002000:00010000:1.0:1789760777.774929:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24387584 -> 24391680 00010000:00010000:1.0:1789760777.774946:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24387584->24391679] (req 24387584->24391679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.774959:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24387584->24391679] (req 24387584->24391679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.774968:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24387584->24391679] (req 24387584->24391679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.774975:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760777.774980:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24387584->24391679] (req 24387584->24391679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.774988:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24387584->24391679] (req 24387584->24391679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.775007:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24387584->24391679] (req 24387584->24391679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.775014:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209c8b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24387584->24391679] (req 24387584->24391679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760777.775248:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484331392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33828us (34071us total) trans 4294973272 rc 0/0 00000100:00100000:1.0:1789760777.775260:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5983 00000100:00100000:1.0:1789760777.775269:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767039860743 time 767072331803 state 0x2 ref 0 00000100:00100000:2.0:1789760777.778641:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760777.778715:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484331520 00000100:00100000:3.0:1789760777.778723:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484331520 00000100:00100000:3.0:1789760777.778777:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767039860743 time 767072331803 state 0x0 ref 1 00000100:00100000:3.0:1789760777.778792:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767072592219 time 767072331803 state 0x1 ref 1 00000100:00100000:3.0:1789760777.778817:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767072592219 time 767109748276 state 0x0 ref 1 00000100:00100000:3.0:1789760777.778944:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484331520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760777.778990:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24391680, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:3.0:1789760777.778998:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.779013:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:3.0:1789760777.779017:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760777.779023:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24391680->24395775] (req 24391680->24395775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.779037:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24391680->24395775] (req 24391680->24395775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760777.806625:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973273 is committed 00002000:00010000:3.0:1789760777.806753:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24391680 -> 24395776 00010000:00010000:3.0:1789760777.806768:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24391680->24395775] (req 24391680->24395775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.806781:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24391680->24395775] (req 24391680->24395775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.806796:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24391680->24395775] (req 24391680->24395775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.806804:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760777.806808:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24391680->24395775] (req 24391680->24395775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.806817:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24391680->24395775] (req 24391680->24395775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.806895:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24391680->24395775] (req 24391680->24395775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.806904:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209c92 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24391680->24395775] (req 24391680->24395775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760777.807023:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5083b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484331520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28192us (28386us total) trans 4294973273 rc 0/0 00000100:00100000:3.0:1789760777.807033:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5984 00000100:00100000:3.0:1789760777.807042:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767072592219 time 767109748276 state 0x2 ref 0 00000100:00100000:3.0:1789760777.810802:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760777.810923:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484331648 00000100:00100000:2.0:1789760777.810933:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484331648 00000100:00100000:2.0:1789760777.810989:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767072592219 time 767109748276 state 0x0 ref 1 00000100:00100000:2.0:1789760777.811003:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767110008692 time 767109748276 state 0x1 ref 1 00000100:00100000:2.0:1789760777.811332:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767110008692 time 767142261505 state 0x0 ref 1 00000100:00100000:2.0:1789760777.811347:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484331648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760777.811397:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24395776, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:2.0:1789760777.811406:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.811421:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:2.0:1789760777.811425:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760777.811431:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24395776->24399871] (req 24395776->24399871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.811450:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24395776->24399871] (req 24395776->24399871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760777.841231:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973274 is committed 00002000:00010000:2.0:1789760777.841301:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24395776 -> 24399872 00010000:00010000:2.0:1789760777.841313:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24395776->24399871] (req 24395776->24399871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.841325:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24395776->24399871] (req 24395776->24399871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.841333:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24395776->24399871] (req 24395776->24399871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.841340:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760777.841344:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24395776->24399871] (req 24395776->24399871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.841353:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24395776->24399871] (req 24395776->24399871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.841373:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24395776->24399871] (req 24395776->24399871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.841380:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209c99 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24395776->24399871] (req 24395776->24399871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760777.841463:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484331648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30114us (30664us total) trans 4294973274 rc 0/0 00000100:00100000:2.0:1789760777.841473:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5985 00000100:00100000:2.0:1789760777.841483:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767110008692 time 767142261505 state 0x2 ref 0 00000100:00100000:2.0:1789760777.844579:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760777.844680:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484331776 00000100:00100000:1.0:1789760777.844689:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484331776 00000100:00100000:1.0:1789760777.844743:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767110008692 time 767142261505 state 0x0 ref 1 00000100:00100000:1.0:1789760777.844758:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767142521921 time 767142261505 state 0x1 ref 1 00000100:00100000:1.0:1789760777.844819:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767142521921 time 767175749616 state 0x0 ref 1 00000100:00100000:1.0:1789760777.844832:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484331776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760777.844883:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24399872, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:1.0:1789760777.844893:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.844909:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:1.0:1789760777.844926:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760777.844934:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24399872->24403967] (req 24399872->24403967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.844950:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24399872->24403967] (req 24399872->24403967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760777.872793:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973275 is committed 00002000:00010000:0.0:1789760777.872944:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24399872 -> 24403968 00010000:00010000:0.0:1789760777.872960:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24399872->24403967] (req 24399872->24403967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.872975:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24399872->24403967] (req 24399872->24403967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.872984:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24399872->24403967] (req 24399872->24403967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.872992:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760777.872997:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24399872->24403967] (req 24399872->24403967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.873008:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24399872->24403967] (req 24399872->24403967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.873029:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24399872->24403967] (req 24399872->24403967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.873037:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209ca0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24399872->24403967] (req 24399872->24403967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760777.873254:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5082a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484331776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28420us (28681us total) trans 4294973275 rc 0/0 00000100:00100000:0.0:1789760777.873265:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5986 00000100:00100000:0.0:1789760777.873273:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767142521921 time 767175749616 state 0x2 ref 0 00000100:00100000:2.0:1789760777.876927:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760777.877017:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484331904 00000100:00100000:1.0:1789760777.877027:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484331904 00000100:00100000:1.0:1789760777.877127:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767142521921 time 767175749616 state 0x0 ref 1 00000100:00100000:1.0:1789760777.877154:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767176010032 time 767175749616 state 0x1 ref 1 00000100:00100000:1.0:1789760777.877186:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767176010032 time 767208116460 state 0x0 ref 1 00000100:00100000:1.0:1789760777.877199:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484331904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760777.877247:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24403968, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:1.0:1789760777.877258:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.877275:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:1.0:1789760777.877279:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760777.877287:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24403968->24408063] (req 24403968->24408063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.877302:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24403968->24408063] (req 24403968->24408063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760777.915258:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24403968 -> 24408064 00010000:00010000:1.0:1789760777.915276:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24403968->24408063] (req 24403968->24408063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.915289:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24403968->24408063] (req 24403968->24408063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.915298:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24403968->24408063] (req 24403968->24408063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.915306:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760777.915311:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24403968->24408063] (req 24403968->24408063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.915319:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24403968->24408063] (req 24403968->24408063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.915337:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24403968->24408063] (req 24403968->24408063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760777.915344:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209ca7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24403968->24408063] (req 24403968->24408063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760777.915443:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484331904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38243us (38520us total) trans 4294973276 rc 0/0 00000100:00100000:1.0:1789760777.915454:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5987 00000100:00100000:1.0:1789760777.915465:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767176010032 time 767208116460 state 0x2 ref 0 00000001:00080000:3.0:1789760777.915541:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973276 is committed 00000100:00100000:3.0:1789760777.918527:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760777.918635:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484332032 00000100:00100000:0.0:1789760777.918645:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484332032 00000100:00100000:0.0:1789760777.918705:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767176010032 time 767208116460 state 0x0 ref 1 00000100:00100000:0.0:1789760777.918721:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767208376876 time 767208116460 state 0x1 ref 1 00000100:00100000:0.0:1789760777.918754:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767208376876 time 767249684915 state 0x0 ref 1 00000100:00100000:0.0:1789760777.918767:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760777.918813:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24408064, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:0.0:1789760777.918821:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.918837:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:0.0:1789760777.918841:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760777.918848:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24408064->24412159] (req 24408064->24412159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760777.918863:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24408064->24412159] (req 24408064->24412159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760777.937834:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973277 is committed 00002000:00010000:3.0:1789760777.937922:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24408064 -> 24412160 00010000:00010000:3.0:1789760777.937935:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24408064->24412159] (req 24408064->24412159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.937944:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24408064->24412159] (req 24408064->24412159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.937950:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24408064->24412159] (req 24408064->24412159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.937954:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760777.937957:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24408064->24412159] (req 24408064->24412159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.937963:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24408064->24412159] (req 24408064->24412159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.937978:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24408064->24412159] (req 24408064->24412159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.937982:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209cae lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24408064->24412159] (req 24408064->24412159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760777.938104:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 19302us (19549us total) trans 4294973277 rc 0/0 00000100:00100000:3.0:1789760777.938114:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5988 00000100:00100000:3.0:1789760777.938123:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767208376876 time 767249684915 state 0x2 ref 0 00000100:00100000:0.0:1789760777.941391:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760777.941785:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484332160 00000100:00100000:2.0:1789760777.941795:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484332160 00000100:00100000:2.0:1789760777.941843:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767208376876 time 767249684915 state 0x0 ref 1 00000100:00100000:2.0:1789760777.941859:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767249945331 time 767249684915 state 0x1 ref 1 00000100:00100000:2.0:1789760777.941890:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767249945331 time 767272821429 state 0x0 ref 1 00000100:00100000:2.0:1789760777.941902:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abb100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484332160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760777.941948:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24412160, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:2.0:1789760777.941957:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.941971:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:2.0:1789760777.941975:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760777.941981:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24412160->24416255] (req 24412160->24416255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.942001:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24412160->24416255] (req 24412160->24416255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760777.967492:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973278 is committed 00002000:00010000:3.0:1789760777.967586:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24412160 -> 24416256 00010000:00010000:3.0:1789760777.967602:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24412160->24416255] (req 24412160->24416255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.967615:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24412160->24416255] (req 24412160->24416255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.967624:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24412160->24416255] (req 24412160->24416255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.967631:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760777.967635:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24412160->24416255] (req 24412160->24416255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.967645:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24412160->24416255] (req 24412160->24416255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.967665:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24412160->24416255] (req 24412160->24416255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760777.967673:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209cb5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24412160->24416255] (req 24412160->24416255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760777.967763:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abb100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484332160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25858us (26378us total) trans 4294973278 rc 0/0 00000100:00100000:3.0:1789760777.967774:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5989 00000100:00100000:3.0:1789760777.967783:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767249945331 time 767272821429 state 0x2 ref 0 00000100:00100000:2.0:1789760777.973586:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760777.974949:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484332288 00000100:00100000:1.0:1789760777.974961:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484332288 00000100:00100000:1.0:1789760777.975021:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767249945331 time 767272821429 state 0x0 ref 1 00000100:00100000:1.0:1789760777.975037:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767273081845 time 767272821429 state 0x1 ref 1 00000100:00100000:3.0:1789760777.975171:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767273081845 time 767306043881 state 0x0 ref 1 00000100:00100000:2.0:1789760777.975247:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a32680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760777.975299:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24416256, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:2.0:1789760777.975307:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.975322:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:2.0:1789760777.975326:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760777.975332:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24416256->24420351] (req 24416256->24420351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760777.975345:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24416256->24420351] (req 24416256->24420351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760778.005264:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973279 is committed 00002000:00010000:3.0:1789760778.005366:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24416256 -> 24420352 00010000:00010000:3.0:1789760778.005381:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24416256->24420351] (req 24416256->24420351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.005394:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24416256->24420351] (req 24416256->24420351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.005404:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24416256->24420351] (req 24416256->24420351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.005412:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760778.005416:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24416256->24420351] (req 24416256->24420351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.005426:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24416256->24420351] (req 24416256->24420351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.005445:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24416256->24420351] (req 24416256->24420351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.005454:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209cbc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24416256->24420351] (req 24416256->24420351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760778.005605:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a32680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30415us (32023us total) trans 4294973279 rc 0/0 00000100:00100000:3.0:1789760778.005616:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5990 00000100:00100000:3.0:1789760778.005626:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767273081845 time 767306043881 state 0x2 ref 0 00000100:00100000:0.0:1789760778.009320:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760778.009425:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484332416 00000100:00100000:1.0:1789760778.009436:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484332416 00000100:00100000:1.0:1789760778.009546:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767273081845 time 767306043881 state 0x0 ref 1 00000100:00100000:1.0:1789760778.009561:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767306304297 time 767306043881 state 0x1 ref 1 00000100:00100000:1.0:1789760778.009619:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767306304297 time 767340549650 state 0x0 ref 1 00000100:00100000:1.0:1789760778.009632:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8aba680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760778.009688:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24420352, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:1.0:1789760778.009698:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.009715:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:1.0:1789760778.009720:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760778.009727:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24420352->24424447] (req 24420352->24424447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.009742:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24420352->24424447] (req 24420352->24424447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760778.010943:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:0.0:1789760778.053298:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973280 is committed 00002000:00010000:1.0:1789760778.053617:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24420352 -> 24424448 00010000:00010000:1.0:1789760778.053633:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24420352->24424447] (req 24420352->24424447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.053646:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24420352->24424447] (req 24420352->24424447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.053656:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24420352->24424447] (req 24420352->24424447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.053663:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760778.053668:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24420352->24424447] (req 24420352->24424447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.053677:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24420352->24424447] (req 24420352->24424447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.053696:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24420352->24424447] (req 24420352->24424447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.053703:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cc3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24420352->24424447] (req 24420352->24424447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760778.053790:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8aba680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44156us (44475us total) trans 4294973280 rc 0/0 00000100:00100000:1.0:1789760778.053800:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5991 00000100:00100000:1.0:1789760778.053810:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767306304297 time 767340549650 state 0x2 ref 0 00000100:00100000:0.0:1789760778.056682:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760778.056747:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484332544 00000100:00100000:1.0:1789760778.056756:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484332544 00000100:00100000:1.0:1789760778.056803:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767306304297 time 767340549650 state 0x0 ref 1 00000100:00100000:1.0:1789760778.056816:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767340810066 time 767340549650 state 0x1 ref 1 00000100:00100000:1.0:1789760778.056923:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767340810066 time 767387853432 state 0x0 ref 1 00000100:00100000:1.0:1789760778.056937:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760778.056981:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24424448, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:1.0:1789760778.056990:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.057005:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:1.0:1789760778.057009:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760778.057015:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24424448->24428543] (req 24424448->24428543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.057028:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24424448->24428543] (req 24424448->24428543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760778.079710:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973281 is committed 00002000:00010000:3.0:1789760778.079870:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24424448 -> 24428544 00010000:00010000:3.0:1789760778.079890:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24424448->24428543] (req 24424448->24428543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.079903:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24424448->24428543] (req 24424448->24428543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.079911:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24424448->24428543] (req 24424448->24428543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.079918:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760778.079923:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24424448->24428543] (req 24424448->24428543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.079932:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24424448->24428543] (req 24424448->24428543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.079952:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24424448->24428543] (req 24424448->24428543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.079960:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209cca lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24424448->24428543] (req 24424448->24428543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760778.080108:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23131us (23393us total) trans 4294973281 rc 0/0 00000100:00100000:3.0:1789760778.080119:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5992 00000100:00100000:3.0:1789760778.080127:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767340810066 time 767387853432 state 0x2 ref 0 00000100:00100000:2.0:1789760778.083054:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760778.083327:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484332672 00000100:00100000:3.0:1789760778.083336:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484332672 00000100:00100000:3.0:1789760778.083385:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767340810066 time 767387853432 state 0x0 ref 1 00000100:00100000:3.0:1789760778.083400:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767388113848 time 767387853432 state 0x1 ref 1 00000100:00100000:3.0:1789760778.083496:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767388113848 time 767414426468 state 0x0 ref 1 00000100:00100000:3.0:1789760778.083509:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760778.083566:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24428544, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:3.0:1789760778.083779:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.083794:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:3.0:1789760778.083797:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760778.083801:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24428544->24432639] (req 24428544->24432639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.083812:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24428544->24432639] (req 24428544->24432639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760778.105715:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973282 is committed 00002000:00010000:3.0:1789760778.105828:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24428544 -> 24432640 00010000:00010000:3.0:1789760778.105845:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24428544->24432639] (req 24428544->24432639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.105857:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24428544->24432639] (req 24428544->24432639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.105866:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24428544->24432639] (req 24428544->24432639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.105874:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760778.105879:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24428544->24432639] (req 24428544->24432639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.105887:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24428544->24432639] (req 24428544->24432639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.105908:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24428544->24432639] (req 24428544->24432639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.105915:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209cd1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24428544->24432639] (req 24428544->24432639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760778.106058:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 22547us (23019us total) trans 4294973282 rc 0/0 00000100:00100000:3.0:1789760778.106113:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5993 00000100:00100000:3.0:1789760778.106123:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767388113848 time 767414426468 state 0x2 ref 0 00000100:00100000:0.0:1789760778.110312:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760778.110552:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484332800 00000100:00100000:3.0:1789760778.110562:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484332800 00000100:00100000:3.0:1789760778.110615:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767388113848 time 767414426468 state 0x0 ref 1 00000100:00100000:3.0:1789760778.110631:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767414686884 time 767414426468 state 0x1 ref 1 00000100:00100000:3.0:1789760778.110709:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767414686884 time 767441638608 state 0x0 ref 1 00000100:00100000:3.0:1789760778.110722:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760778.110771:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24432640, mtime: 0x6aad9509, atime: 0x0, ctime: 0x6aad9509, blocks: 0xc007 00010000:00010000:3.0:1789760778.110780:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.110795:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:3.0:1789760778.110799:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760778.110804:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24432640->24436735] (req 24432640->24436735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.110816:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24432640->24436735] (req 24432640->24436735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.127478:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760778.151516:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973283 is committed 00002000:00010000:0.0:1789760778.151644:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24432640 -> 24436736 00002000:00010000:0.0:1789760778.151650:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760777 -> 1789760778 00002000:00010000:0.0:1789760778.151652:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760777 -> 1789760778 00010000:00010000:0.0:1789760778.151665:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24432640->24436735] (req 24432640->24436735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.151679:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24432640->24436735] (req 24432640->24436735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.151690:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24432640->24436735] (req 24432640->24436735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.151696:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760778.151701:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24432640->24436735] (req 24432640->24436735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.151709:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24432640->24436735] (req 24432640->24436735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.151728:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24432640->24436735] (req 24432640->24436735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.151736:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209cd8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24432640->24436735] (req 24432640->24436735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760778.152000:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41192us (41610us total) trans 4294973283 rc 0/0 00000100:00100000:0.0:1789760778.152012:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5994 00000100:00100000:0.0:1789760778.152021:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767414686884 time 767441638608 state 0x2 ref 0 00000100:00100000:0.0:1789760778.156231:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760778.156267:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484332928 00000100:00100000:1.0:1789760778.156279:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484332928 00000100:00100000:1.0:1789760778.156408:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767414686884 time 767441638608 state 0x0 ref 1 00000100:00100000:1.0:1789760778.156423:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767441899024 time 767441638608 state 0x1 ref 1 00000100:00100000:1.0:1789760778.156449:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767441899024 time 767487379799 state 0x0 ref 1 00000100:00100000:1.0:1789760778.156462:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760778.156515:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24436736, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:1.0:1789760778.156524:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.156540:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:1.0:1789760778.156544:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760778.156550:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24436736->24440831] (req 24436736->24440831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.156565:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24436736->24440831] (req 24436736->24440831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760778.202670:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973284 is committed 00002000:00010000:0.0:1789760778.202969:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24436736 -> 24440832 00010000:00010000:0.0:1789760778.202986:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24436736->24440831] (req 24436736->24440831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.203006:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24436736->24440831] (req 24436736->24440831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.203015:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24436736->24440831] (req 24436736->24440831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.203023:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760778.203027:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24436736->24440831] (req 24436736->24440831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.203037:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24436736->24440831] (req 24436736->24440831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.203138:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24436736->24440831] (req 24436736->24440831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.203147:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209cdf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24436736->24440831] (req 24436736->24440831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760778.203470:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484332928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47004us (47242us total) trans 4294973284 rc 0/0 00000100:00100000:0.0:1789760778.203481:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5995 00000100:00100000:0.0:1789760778.203491:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767441899024 time 767487379799 state 0x2 ref 0 00000100:00100000:3.0:1789760778.208512:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760778.208641:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484333056 00000100:00100000:1.0:1789760778.208651:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484333056 00000100:00100000:1.0:1789760778.208698:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767441899024 time 767487379799 state 0x0 ref 1 00000100:00100000:1.0:1789760778.208726:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767487640215 time 767487379799 state 0x1 ref 1 00000100:00100000:1.0:1789760778.208754:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767487640215 time 767539685458 state 0x0 ref 1 00000100:00100000:1.0:1789760778.208767:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484333056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760778.208814:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24440832, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:1.0:1789760778.208823:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.208850:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:1.0:1789760778.208855:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760778.208862:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24440832->24444927] (req 24440832->24444927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.208876:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24440832->24444927] (req 24440832->24444927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760778.240537:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973285 is committed 00002000:00010000:1.0:1789760778.240635:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24440832 -> 24444928 00010000:00010000:1.0:1789760778.240652:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24440832->24444927] (req 24440832->24444927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.240666:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24440832->24444927] (req 24440832->24444927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.240676:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24440832->24444927] (req 24440832->24444927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.240688:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760778.240693:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24440832->24444927] (req 24440832->24444927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.240701:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24440832->24444927] (req 24440832->24444927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.240721:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24440832->24444927] (req 24440832->24444927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.240729:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209ce6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24440832->24444927] (req 24440832->24444927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760778.241156:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484333056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32386us (32648us total) trans 4294973285 rc 0/0 00000100:00100000:1.0:1789760778.241167:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5996 00000100:00100000:1.0:1789760778.241176:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767487640215 time 767539685458 state 0x2 ref 0 00000100:00100000:3.0:1789760778.257811:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760778.260661:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484333184 00000100:00100000:2.0:1789760778.260676:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484333184 00000100:00100000:2.0:1789760778.260736:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767487640215 time 767539685458 state 0x0 ref 1 00000100:00100000:2.0:1789760778.260753:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767539945874 time 767539685458 state 0x1 ref 1 00000100:00100000:2.0:1789760778.260775:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767539945874 time 767591707235 state 0x0 ref 1 00000100:00100000:2.0:1789760778.260788:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484333184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760778.260834:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24444928, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:2.0:1789760778.260843:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.260859:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:2.0:1789760778.260864:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760778.260870:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24444928->24449023] (req 24444928->24449023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.260884:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24444928->24449023] (req 24444928->24449023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760778.311600:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973286 is committed 00002000:00010000:0.0:1789760778.312233:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24444928 -> 24449024 00010000:00010000:0.0:1789760778.312251:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24444928->24449023] (req 24444928->24449023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.312264:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24444928->24449023] (req 24444928->24449023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.312272:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24444928->24449023] (req 24444928->24449023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.312279:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760778.312284:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24444928->24449023] (req 24444928->24449023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.312291:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24444928->24449023] (req 24444928->24449023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.312314:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24444928->24449023] (req 24444928->24449023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.312321:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ced lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24444928->24449023] (req 24444928->24449023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760778.312415:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484333184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51625us (54608us total) trans 4294973286 rc 0/0 00000100:00100000:0.0:1789760778.312425:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5997 00000100:00100000:0.0:1789760778.312435:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767539945874 time 767591707235 state 0x2 ref 0 00000100:00100000:0.0:1789760778.316355:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760778.316478:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484333312 00000100:00100000:3.0:1789760778.316488:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484333312 00000100:00100000:3.0:1789760778.316542:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767539945874 time 767591707235 state 0x0 ref 1 00000100:00100000:3.0:1789760778.316557:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767591967651 time 767591707235 state 0x1 ref 1 00000100:00100000:3.0:1789760778.316586:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767591967651 time 767647516680 state 0x0 ref 1 00000100:00100000:3.0:1789760778.316598:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484333312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760778.316646:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24449024, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:3.0:1789760778.316656:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.316672:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:3.0:1789760778.316676:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760778.316683:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24449024->24453119] (req 24449024->24453119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.316696:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24449024->24453119] (req 24449024->24453119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760778.351058:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973287 is committed 00002000:00010000:0.0:1789760778.351374:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24449024 -> 24453120 00010000:00010000:0.0:1789760778.351389:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24449024->24453119] (req 24449024->24453119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.351401:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24449024->24453119] (req 24449024->24453119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.351410:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24449024->24453119] (req 24449024->24453119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.351417:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760778.351422:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24449024->24453119] (req 24449024->24453119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.351430:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24449024->24453119] (req 24449024->24453119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.351449:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24449024->24453119] (req 24449024->24453119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.351457:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209cf4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24449024->24453119] (req 24449024->24453119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760778.351584:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484333312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34984us (35234us total) trans 4294973287 rc 0/0 00000100:00100000:0.0:1789760778.351596:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5998 00000100:00100000:0.0:1789760778.351605:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767591967651 time 767647516680 state 0x2 ref 0 00000100:00100000:2.0:1789760778.364264:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760778.364330:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484333440 00000100:00100000:0.0:1789760778.364342:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484333440 00000100:00100000:0.0:1789760778.364405:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767591967651 time 767647516680 state 0x0 ref 1 00000100:00100000:0.0:1789760778.364422:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767647777096 time 767647516680 state 0x1 ref 1 00000100:00100000:0.0:1789760778.364455:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767647777096 time 767695385849 state 0x0 ref 1 00000100:00100000:0.0:1789760778.364469:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a32d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484333440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760778.364519:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24453120, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:0.0:1789760778.364528:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.364544:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:0.0:1789760778.364549:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760778.364555:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24453120->24457215] (req 24453120->24457215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.364570:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24453120->24457215] (req 24453120->24457215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760778.401724:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24453120 -> 24457216 00010000:00010000:0.0:1789760778.401746:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24453120->24457215] (req 24453120->24457215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.401759:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24453120->24457215] (req 24453120->24457215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.401769:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24453120->24457215] (req 24453120->24457215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.401777:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760778.401782:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24453120->24457215] (req 24453120->24457215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.401792:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24453120->24457215] (req 24453120->24457215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.401813:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24453120->24457215] (req 24453120->24457215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.402230:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209cfb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24453120->24457215] (req 24453120->24457215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760778.402369:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a32d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484333440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37899us (38111us total) trans 4294973288 rc 0/0 00000100:00100000:0.0:1789760778.402380:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 5999 00000100:00100000:0.0:1789760778.402389:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767647777096 time 767695385849 state 0x2 ref 0 00000001:00080000:2.0:1789760778.402479:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973288 is committed 00000100:00100000:2.0:1789760778.406284:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760778.406379:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484333568 00000100:00100000:1.0:1789760778.406388:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484333568 00000100:00100000:1.0:1789760778.406590:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767647777096 time 767695385849 state 0x0 ref 1 00000100:00100000:1.0:1789760778.406609:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767695646265 time 767695385849 state 0x1 ref 1 00000100:00100000:1.0:1789760778.406675:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767695646265 time 767737605135 state 0x0 ref 1 00000100:00100000:1.0:1789760778.406692:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484333568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760778.406741:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24457216, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:1.0:1789760778.406751:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.406767:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:1.0:1789760778.406772:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760778.406777:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24457216->24461311] (req 24457216->24461311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.406792:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24457216->24461311] (req 24457216->24461311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760778.444016:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973289 is committed 00002000:00010000:1.0:1789760778.444312:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24457216 -> 24461312 00010000:00010000:1.0:1789760778.444327:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24457216->24461311] (req 24457216->24461311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.444340:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24457216->24461311] (req 24457216->24461311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.444349:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24457216->24461311] (req 24457216->24461311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.444356:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760778.444360:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24457216->24461311] (req 24457216->24461311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.444370:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24457216->24461311] (req 24457216->24461311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.444390:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24457216->24461311] (req 24457216->24461311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.444398:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209d02 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24457216->24461311] (req 24457216->24461311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760778.444777:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484333568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38082us (38499us total) trans 4294973289 rc 0/0 00000100:00100000:1.0:1789760778.444788:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6000 00000100:00100000:1.0:1789760778.444799:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767695646265 time 767737605135 state 0x2 ref 0 00000100:00100000:1.0:1789760778.452194:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760778.452384:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484333696 00000100:00100000:2.0:1789760778.452393:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484333696 00000100:00100000:2.0:1789760778.452453:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767695646265 time 767737605135 state 0x0 ref 1 00000100:00100000:2.0:1789760778.452469:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767737865551 time 767737605135 state 0x1 ref 1 00000100:00100000:2.0:1789760778.452498:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767737865551 time 767783428894 state 0x0 ref 1 00000100:00100000:2.0:1789760778.452509:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484333696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760778.452557:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24461312, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:2.0:1789760778.452566:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.452582:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:2.0:1789760778.452586:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760778.452593:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24461312->24465407] (req 24461312->24465407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.452606:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24461312->24465407] (req 24461312->24465407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760778.479676:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760778.479750:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760778.479841:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760778.479858:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760778.479976:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484333952 00000100:00100000:2.0:1789760778.479985:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484333952 00000100:00100000:2.0:1789760778.480051:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484334208 00000100:00100000:2.0:1789760778.480055:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484334208 00000100:00100000:2.0:1789760778.480109:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 991 00000100:00100000:2.0:1789760778.480118:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2422:x1876699484333952:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760778.480191:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2422:x1876699484333952:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 73us (521us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760778.480198:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 991 00000100:00100000:2.0:1789760778.480214:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 992 00000100:00100000:2.0:1789760778.480220:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f13480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484334208:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760778.480245:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f13480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484334208:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 25us (495us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760778.480252:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 992 00000100:00080000:2.0:1789760778.480270:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:1.0:1789760778.480907:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760778.481138:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484334080 00000100:00100000:3.0:1789760778.481145:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484334080 00000100:00100000:3.0:1789760778.481179:0:6474:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484334336 00000100:00100000:3.0:1789760778.481182:0:6474:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484334336 00000100:00100000:3.0:1789760778.481202:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 993 00000100:00100000:3.0:1789760778.481210:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484334080:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760778.481255:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484334080:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 46us (1395us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760778.481262:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 993 00000100:00100000:3.0:1789760778.481275:0:6474:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 994 00000100:00100000:3.0:1789760778.481282:0:6474:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484334336:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760778.481306:0:6474:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2422:x1876699484334336:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 25us (401us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760778.481312:0:6474:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 994 00000100:00080000:0.0:1789760778.481316:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:3.0:1789760778.481328:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484333824 00000100:00100000:3.0:1789760778.481332:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484333824 00000100:00100000:3.0:1789760778.481356:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 493 00000100:00100000:3.0:1789760778.481362:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c803a450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484333824:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:3.0:1789760778.481385:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c803a450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2422:x1876699484333824:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 23us (1543us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760778.481391:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 493 00000001:00080000:1.0:1789760778.502838:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973290 is committed 00002000:00010000:3.0:1789760778.502902:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24461312 -> 24465408 00010000:00010000:3.0:1789760778.502915:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24461312->24465407] (req 24461312->24465407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.502925:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24461312->24465407] (req 24461312->24465407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.502931:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24461312->24465407] (req 24461312->24465407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.502942:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760778.502947:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24461312->24465407] (req 24461312->24465407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.502956:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24461312->24465407] (req 24461312->24465407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.502976:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24461312->24465407] (req 24461312->24465407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.502985:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209d09 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24461312->24465407] (req 24461312->24465407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760778.503151:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484333696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50638us (50962us total) trans 4294973290 rc 0/0 00000100:00100000:3.0:1789760778.503161:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6001 00000100:00100000:3.0:1789760778.503170:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767737865551 time 767783428894 state 0x2 ref 0 00000100:00100000:1.0:1789760778.507363:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760778.507657:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484334464 00000100:00100000:0.0:1789760778.507666:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484334464 00000100:00100000:0.0:1789760778.507728:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767737865551 time 767783428894 state 0x0 ref 1 00000100:00100000:0.0:1789760778.507745:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767783689310 time 767783428894 state 0x1 ref 1 00000100:00100000:0.0:1789760778.507781:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767783689310 time 767838711893 state 0x0 ref 1 00000100:00100000:0.0:1789760778.507795:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484334464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760778.507846:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24465408, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:0.0:1789760778.507855:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.507871:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:0.0:1789760778.507876:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760778.507882:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24465408->24469503] (req 24465408->24469503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.507896:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24465408->24469503] (req 24465408->24469503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760778.541188:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973291 is committed 00002000:00010000:3.0:1789760778.541275:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24465408 -> 24469504 00010000:00010000:3.0:1789760778.541291:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24465408->24469503] (req 24465408->24469503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.541305:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24465408->24469503] (req 24465408->24469503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.541321:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24465408->24469503] (req 24465408->24469503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.541330:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760778.541335:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24465408->24469503] (req 24465408->24469503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.541345:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24465408->24469503] (req 24465408->24469503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.541367:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24465408->24469503] (req 24465408->24469503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.541375:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209d10 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24465408->24469503] (req 24465408->24469503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760778.542153:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484334464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34354us (34793us total) trans 4294973291 rc 0/0 00000100:00100000:3.0:1789760778.542165:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6002 00000100:00100000:3.0:1789760778.542176:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767783689310 time 767838711893 state 0x2 ref 0 00000100:00100000:2.0:1789760778.546802:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760778.546991:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484334592 00000100:00100000:0.0:1789760778.547001:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484334592 00000100:00100000:0.0:1789760778.547097:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767783689310 time 767838711893 state 0x0 ref 1 00000100:00100000:0.0:1789760778.547112:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767838972309 time 767838711893 state 0x1 ref 1 00000100:00100000:0.0:1789760778.547140:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767838972309 time 767878071970 state 0x0 ref 1 00000100:00100000:0.0:1789760778.547153:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484334592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760778.547200:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24469504, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:0.0:1789760778.547209:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.547225:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:0.0:1789760778.547229:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760778.547235:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24469504->24473599] (req 24469504->24473599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.547249:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24469504->24473599] (req 24469504->24473599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760778.575919:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24469504 -> 24473600 00010000:00010000:0.0:1789760778.575938:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24469504->24473599] (req 24469504->24473599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.575950:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24469504->24473599] (req 24469504->24473599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.575960:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24469504->24473599] (req 24469504->24473599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.575967:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760778.575972:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24469504->24473599] (req 24469504->24473599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.575980:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24469504->24473599] (req 24469504->24473599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.576002:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24469504->24473599] (req 24469504->24473599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.576010:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d17 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24469504->24473599] (req 24469504->24473599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760778.576234:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484334592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29078us (29435us total) trans 4294973292 rc 0/0 00000100:00100000:0.0:1789760778.576245:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6003 00000100:00100000:0.0:1789760778.576255:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767838972309 time 767878071970 state 0x2 ref 0 00000001:00080000:0.0:1789760778.576295:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973292 is committed 00000100:00100000:1.0:1789760778.579761:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760778.580056:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484334720 00000100:00100000:3.0:1789760778.580258:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484334720 00000100:00100000:3.0:1789760778.580379:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767838972309 time 767878071970 state 0x0 ref 1 00000100:00100000:3.0:1789760778.580395:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767878332386 time 767878071970 state 0x1 ref 1 00000100:00100000:3.0:1789760778.580465:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767878332386 time 767911395688 state 0x0 ref 1 00000100:00100000:3.0:1789760778.580479:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484334720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760778.580527:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24473600, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:3.0:1789760778.580535:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.580550:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:3.0:1789760778.580553:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760778.580559:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24473600->24477695] (req 24473600->24477695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.580573:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24473600->24477695] (req 24473600->24477695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760778.610136:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973293 is committed 00002000:00010000:0.0:1789760778.610353:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24473600 -> 24477696 00010000:00010000:0.0:1789760778.610369:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24473600->24477695] (req 24473600->24477695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.610382:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24473600->24477695] (req 24473600->24477695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.610390:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24473600->24477695] (req 24473600->24477695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.610403:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760778.610472:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24473600->24477695] (req 24473600->24477695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.610483:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24473600->24477695] (req 24473600->24477695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.610504:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24473600->24477695] (req 24473600->24477695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.610512:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209d1e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24473600->24477695] (req 24473600->24477695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760778.610687:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484334720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30205us (30928us total) trans 4294973293 rc 0/0 00000100:00100000:0.0:1789760778.610697:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6004 00000100:00100000:0.0:1789760778.610707:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767878332386 time 767911395688 state 0x2 ref 0 00000100:00100000:3.0:1789760778.615310:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760778.615901:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484334848 00000100:00100000:2.0:1789760778.615912:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484334848 00000100:00100000:2.0:1789760778.615968:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767878332386 time 767911395688 state 0x0 ref 1 00000100:00100000:2.0:1789760778.615984:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767911656104 time 767911395688 state 0x1 ref 1 00000100:00100000:2.0:1789760778.616017:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767911656104 time 767946947760 state 0x0 ref 1 00000100:00100000:2.0:1789760778.616030:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484334848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760778.616127:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24477696, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:2.0:1789760778.616137:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.616152:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:2.0:1789760778.616157:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760778.616163:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24477696->24481791] (req 24477696->24481791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.616176:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24477696->24481791] (req 24477696->24481791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760778.652227:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973294 is committed 00002000:00010000:3.0:1789760778.652428:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24477696 -> 24481792 00010000:00010000:3.0:1789760778.652442:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24477696->24481791] (req 24477696->24481791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.652454:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24477696->24481791] (req 24477696->24481791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.652463:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24477696->24481791] (req 24477696->24481791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.652476:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760778.652482:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24477696->24481791] (req 24477696->24481791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.652491:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24477696->24481791] (req 24477696->24481791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.652739:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24477696->24481791] (req 24477696->24481791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.652747:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000025f69bce/0x2cf3826396209d25 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24477696->24481791] (req 24477696->24481791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760778.653320:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484334848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37286us (38022us total) trans 4294973294 rc 0/0 00000100:00100000:3.0:1789760778.653330:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6005 00000100:00100000:3.0:1789760778.653340:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767911656104 time 767946947760 state 0x2 ref 0 00000100:00100000:2.0:1789760778.659320:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760778.662163:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484334976 00000100:00100000:0.0:1789760778.662178:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484334976 00000100:00100000:0.0:1789760778.662249:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767911656104 time 767946947760 state 0x0 ref 1 00000100:00100000:0.0:1789760778.662267:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767947208176 time 767946947760 state 0x1 ref 1 00000100:00100000:0.0:1789760778.662305:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767947208176 time 767993235473 state 0x0 ref 1 00000100:00100000:0.0:1789760778.662323:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484334976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760778.662371:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24481792, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:0.0:1789760778.662379:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.662395:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=767s v=(0 0) 00010000:00010000:0.0:1789760778.662400:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760778.662406:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24481792->24485887] (req 24481792->24485887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.662418:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24481792->24485887] (req 24481792->24485887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760778.720577:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973295 is committed 00002000:00010000:1.0:1789760778.720681:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24481792 -> 24485888 00010000:00010000:1.0:1789760778.720696:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24481792->24485887] (req 24481792->24485887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.720711:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24481792->24485887] (req 24481792->24485887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.720720:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24481792->24485887] (req 24481792->24485887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.720727:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760778.720731:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24481792->24485887] (req 24481792->24485887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.720740:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24481792->24485887] (req 24481792->24485887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.720763:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24481792->24485887] (req 24481792->24485887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.720770:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209d2c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24481792->24485887] (req 24481792->24485887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760778.720875:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484334976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58552us (61560us total) trans 4294973295 rc 0/0 00000100:00100000:1.0:1789760778.720886:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6006 00000100:00100000:1.0:1789760778.720896:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767947208176 time 767993235473 state 0x2 ref 0 00000100:00100000:1.0:1789760778.723622:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760778.723705:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484335104 00000100:00100000:0.0:1789760778.723714:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484335104 00000100:00100000:0.0:1789760778.723769:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767947208176 time 767993235473 state 0x0 ref 1 00000100:00100000:0.0:1789760778.723784:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767993495889 time 767993235473 state 0x1 ref 1 00000100:00100000:0.0:1789760778.723811:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767993495889 time 768054742817 state 0x0 ref 1 00000100:00100000:0.0:1789760778.723834:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484335104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760778.723879:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24485888, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:0.0:1789760778.723887:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.723908:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:0.0:1789760778.723912:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760778.723918:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24485888->24489983] (req 24485888->24489983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760778.723932:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24485888->24489983] (req 24485888->24489983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760778.755729:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973296 is committed 00002000:00010000:2.0:1789760778.755809:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24485888 -> 24489984 00010000:00010000:2.0:1789760778.755891:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24485888->24489983] (req 24485888->24489983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.755903:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24485888->24489983] (req 24485888->24489983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.755912:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24485888->24489983] (req 24485888->24489983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.755917:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760778.755920:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24485888->24489983] (req 24485888->24489983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.755926:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24485888->24489983] (req 24485888->24489983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.755943:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24485888->24489983] (req 24485888->24489983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.755947:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d33 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24485888->24489983] (req 24485888->24489983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760778.756047:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484335104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32213us (32422us total) trans 4294973296 rc 0/0 00000100:00100000:2.0:1789760778.756056:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6007 00000100:00100000:2.0:1789760778.756100:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767993495889 time 768054742817 state 0x2 ref 0 00000100:00100000:1.0:1789760778.760730:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760778.760847:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484335232 00000100:00100000:3.0:1789760778.760857:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484335232 00000100:00100000:3.0:1789760778.760906:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 767993495889 time 768054742817 state 0x0 ref 1 00000100:00100000:3.0:1789760778.760917:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768055003233 time 768054742817 state 0x1 ref 1 00000100:00100000:3.0:1789760778.760944:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768055003233 time 768091874359 state 0x0 ref 1 00000100:00100000:3.0:1789760778.760999:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484335232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760778.761061:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24489984, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:3.0:1789760778.761217:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.761429:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:3.0:1789760778.761433:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760778.761440:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24489984->24494079] (req 24489984->24494079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.761454:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24489984->24494079] (req 24489984->24494079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760778.794712:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973297 is committed 00002000:00010000:1.0:1789760778.794999:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24489984 -> 24494080 00010000:00010000:1.0:1789760778.795015:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24489984->24494079] (req 24489984->24494079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.795029:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24489984->24494079] (req 24489984->24494079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.795132:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24489984->24494079] (req 24489984->24494079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.795141:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760778.795146:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24489984->24494079] (req 24489984->24494079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.795156:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24489984->24494079] (req 24489984->24494079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.795177:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24489984->24494079] (req 24489984->24494079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.795185:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209d3a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24489984->24494079] (req 24489984->24494079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760778.795281:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484335232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34321us (34556us total) trans 4294973297 rc 0/0 00000100:00100000:1.0:1789760778.795291:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6008 00000100:00100000:1.0:1789760778.795301:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768055003233 time 768091874359 state 0x2 ref 0 00000100:00100000:0.0:1789760778.799418:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760778.799905:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484335360 00000100:00100000:3.0:1789760778.799915:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484335360 00000100:00100000:3.0:1789760778.799969:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768055003233 time 768091874359 state 0x0 ref 1 00000100:00100000:3.0:1789760778.799989:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768092134775 time 768091874359 state 0x1 ref 1 00000100:00100000:3.0:1789760778.800022:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768092134775 time 768130953700 state 0x0 ref 1 00000100:00100000:3.0:1789760778.803877:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484335360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760778.803963:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24494080, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:3.0:1789760778.803972:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.803989:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:3.0:1789760778.803994:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760778.804000:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24494080->24498175] (req 24494080->24498175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.804017:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24494080->24498175] (req 24494080->24498175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760778.834086:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973298 is committed 00002000:00010000:3.0:1789760778.834191:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24494080 -> 24498176 00010000:00010000:3.0:1789760778.834206:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24494080->24498175] (req 24494080->24498175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.834220:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24494080->24498175] (req 24494080->24498175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.834229:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24494080->24498175] (req 24494080->24498175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.834236:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760778.834241:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24494080->24498175] (req 24494080->24498175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.834251:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24494080->24498175] (req 24494080->24498175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.834271:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24494080->24498175] (req 24494080->24498175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.834279:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209d41 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24494080->24498175] (req 24494080->24498175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760778.834406:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484335360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34368us (34992us total) trans 4294973298 rc 0/0 00000100:00100000:3.0:1789760778.834416:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6009 00000100:00100000:3.0:1789760778.834426:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768092134775 time 768130953700 state 0x2 ref 0 00000100:00100000:3.0:1789760778.837671:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760778.837784:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484335488 00000100:00100000:1.0:1789760778.837795:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484335488 00000100:00100000:1.0:1789760778.837848:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768092134775 time 768130953700 state 0x0 ref 1 00000100:00100000:1.0:1789760778.837863:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768131214116 time 768130953700 state 0x1 ref 1 00000100:00100000:1.0:1789760778.837923:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768131214116 time 768168853497 state 0x0 ref 1 00000100:00100000:1.0:1789760778.837936:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7069450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484335488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760778.837985:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24498176, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:1.0:1789760778.837999:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.838017:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:1.0:1789760778.838022:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760778.838028:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24498176->24502271] (req 24498176->24502271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760778.838058:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24498176->24502271] (req 24498176->24502271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760778.874414:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973299 is committed 00002000:00010000:3.0:1789760778.874675:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24498176 -> 24502272 00010000:00010000:3.0:1789760778.874690:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24498176->24502271] (req 24498176->24502271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.874702:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24498176->24502271] (req 24498176->24502271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.874712:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24498176->24502271] (req 24498176->24502271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.874719:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760778.874725:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24498176->24502271] (req 24498176->24502271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.874740:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24498176->24502271] (req 24498176->24502271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.874760:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24498176->24502271] (req 24498176->24502271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760778.874768:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209d48 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24498176->24502271] (req 24498176->24502271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760778.874865:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7069450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484335488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36926us (37198us total) trans 4294973299 rc 0/0 00000100:00100000:3.0:1789760778.874875:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6010 00000100:00100000:3.0:1789760778.874883:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768131214116 time 768168853497 state 0x2 ref 0 00000100:00100000:1.0:1789760778.881310:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760778.881834:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484335616 00000100:00100000:2.0:1789760778.881844:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484335616 00000100:00100000:2.0:1789760778.881895:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768131214116 time 768168853497 state 0x0 ref 1 00000100:00100000:2.0:1789760778.881908:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768169113913 time 768168853497 state 0x1 ref 1 00000100:00100000:2.0:1789760778.881937:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768169113913 time 768212868528 state 0x0 ref 1 00000100:00100000:2.0:1789760778.881949:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484335616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760778.881994:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24502272, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:2.0:1789760778.882003:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.882018:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:2.0:1789760778.882022:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760778.882028:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24502272->24506367] (req 24502272->24506367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.882262:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24502272->24506367] (req 24502272->24506367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760778.934580:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973300 is committed 00002000:00010000:2.0:1789760778.934691:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24502272 -> 24506368 00010000:00010000:2.0:1789760778.934706:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24502272->24506367] (req 24502272->24506367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.934724:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24502272->24506367] (req 24502272->24506367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.934732:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24502272->24506367] (req 24502272->24506367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.934738:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760778.934742:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24502272->24506367] (req 24502272->24506367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.934751:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24502272->24506367] (req 24502272->24506367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.934770:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24502272->24506367] (req 24502272->24506367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.934777:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d4f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24502272->24506367] (req 24502272->24506367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760778.934938:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484335616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52985us (53632us total) trans 4294973300 rc 0/0 00000100:00100000:2.0:1789760778.934948:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6011 00000100:00100000:2.0:1789760778.934958:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768169113913 time 768212868528 state 0x2 ref 0 00000100:00100000:3.0:1789760778.938434:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760778.938594:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484335744 00000100:00100000:2.0:1789760778.938605:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484335744 00000100:00100000:2.0:1789760778.938662:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768169113913 time 768212868528 state 0x0 ref 1 00000100:00100000:2.0:1789760778.938677:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768213128944 time 768212868528 state 0x1 ref 1 00000100:00100000:2.0:1789760778.938710:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768213128944 time 768269639978 state 0x0 ref 1 00000100:00100000:2.0:1789760778.938723:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484335744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760778.938772:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24506368, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:2.0:1789760778.938780:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.938794:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:2.0:1789760778.938798:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760778.938805:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24506368->24510463] (req 24506368->24510463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.938931:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24506368->24510463] (req 24506368->24510463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760778.971312:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973301 is committed 00002000:00010000:2.0:1789760778.971417:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24506368 -> 24510464 00010000:00010000:2.0:1789760778.971433:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24506368->24510463] (req 24506368->24510463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.971445:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24506368->24510463] (req 24506368->24510463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.971460:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24506368->24510463] (req 24506368->24510463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.971468:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760778.971473:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24506368->24510463] (req 24506368->24510463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.971481:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24506368->24510463] (req 24506368->24510463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.971500:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24506368->24510463] (req 24506368->24510463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.971509:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209d56 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24506368->24510463] (req 24506368->24510463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760778.971607:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484335744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32882us (33178us total) trans 4294973301 rc 0/0 00000100:00100000:2.0:1789760778.971617:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6012 00000100:00100000:2.0:1789760778.971626:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768213128944 time 768269639978 state 0x2 ref 0 00000100:00100000:3.0:1789760778.975198:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760778.975317:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484335872 00000100:00100000:2.0:1789760778.975328:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484335872 00000100:00100000:2.0:1789760778.975382:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768213128944 time 768269639978 state 0x0 ref 1 00000100:00100000:2.0:1789760778.975398:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768269900394 time 768269639978 state 0x1 ref 1 00000100:00100000:2.0:1789760778.975422:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768269900394 time 768306352818 state 0x0 ref 1 00000100:00100000:2.0:1789760778.975434:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484335872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760778.975482:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24510464, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:2.0:1789760778.975492:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.975507:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:2.0:1789760778.975511:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760778.975517:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24510464->24514559] (req 24510464->24514559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760778.975531:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24510464->24514559] (req 24510464->24514559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760779.006869:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973302 is committed 00002000:00010000:2.0:1789760779.007032:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24510464 -> 24514560 00010000:00010000:2.0:1789760779.007058:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24510464->24514559] (req 24510464->24514559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.007102:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24510464->24514559] (req 24510464->24514559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.007112:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24510464->24514559] (req 24510464->24514559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.007119:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760779.007134:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24510464->24514559] (req 24510464->24514559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.007144:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24510464->24514559] (req 24510464->24514559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.007171:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24510464->24514559] (req 24510464->24514559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.007180:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d5d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24510464->24514559] (req 24510464->24514559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760779.007269:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484335872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31832us (32075us total) trans 4294973302 rc 0/0 00000100:00100000:2.0:1789760779.007281:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6013 00000100:00100000:2.0:1789760779.007297:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768269900394 time 768306352818 state 0x2 ref 0 00000100:00100000:3.0:1789760779.012139:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760779.012469:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484336000 00000100:00100000:0.0:1789760779.012479:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484336000 00000100:00100000:0.0:1789760779.012538:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768269900394 time 768306352818 state 0x0 ref 1 00000100:00100000:0.0:1789760779.012554:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768306613234 time 768306352818 state 0x1 ref 1 00000100:00100000:0.0:1789760779.012585:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768306613234 time 768343515969 state 0x0 ref 1 00000100:00100000:0.0:1789760779.012598:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484336000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760779.012648:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24514560, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:0.0:1789760779.012658:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.012674:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:0.0:1789760779.012723:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760779.012729:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24514560->24518655] (req 24514560->24518655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.012744:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24514560->24518655] (req 24514560->24518655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760779.044293:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24514560 -> 24518656 00010000:00010000:2.0:1789760779.044314:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24514560->24518655] (req 24514560->24518655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.044329:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24514560->24518655] (req 24514560->24518655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.044338:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24514560->24518655] (req 24514560->24518655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.044350:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760779.044355:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24514560->24518655] (req 24514560->24518655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.044364:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24514560->24518655] (req 24514560->24518655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.044384:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24514560->24518655] (req 24514560->24518655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.044392:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d64 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24514560->24518655] (req 24514560->24518655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760779.044486:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484336000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31887us (32364us total) trans 4294973303 rc 0/0 00000100:00100000:2.0:1789760779.044495:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6014 00000100:00100000:2.0:1789760779.044503:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768306613234 time 768343515969 state 0x2 ref 0 00000001:00080000:3.0:1789760779.044572:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973303 is committed 00000100:00100000:0.0:1789760779.054755:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760779.056175:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484336128 00000100:00100000:1.0:1789760779.056188:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484336128 00000100:00100000:1.0:1789760779.056248:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768306613234 time 768343515969 state 0x0 ref 1 00000100:00100000:1.0:1789760779.056265:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768343776385 time 768343515969 state 0x1 ref 1 00000100:00100000:1.0:1789760779.056302:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768343776385 time 768387232743 state 0x0 ref 1 00000100:00100000:1.0:1789760779.056315:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484336128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760779.056378:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24518656, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:1.0:1789760779.056387:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.056403:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:1.0:1789760779.056408:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760779.056415:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24518656->24522751] (req 24518656->24522751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.056428:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24518656->24522751] (req 24518656->24522751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760779.106885:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973304 is committed 00002000:00010000:2.0:1789760779.107029:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24518656 -> 24522752 00010000:00010000:2.0:1789760779.107058:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24518656->24522751] (req 24518656->24522751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.107488:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24518656->24522751] (req 24518656->24522751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.107504:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24518656->24522751] (req 24518656->24522751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.107512:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760779.107517:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24518656->24522751] (req 24518656->24522751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.107526:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24518656->24522751] (req 24518656->24522751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.107546:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24518656->24522751] (req 24518656->24522751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.107554:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209d6b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24518656->24522751] (req 24518656->24522751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760779.107903:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484336128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51581us (53147us total) trans 4294973304 rc 0/0 00000100:00100000:3.0:1789760779.107915:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6015 00000100:00100000:3.0:1789760779.107924:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768343776385 time 768387232743 state 0x2 ref 0 00000100:00100000:1.0:1789760779.110644:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760779.110735:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484336256 00000100:00100000:2.0:1789760779.110745:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484336256 00000100:00100000:2.0:1789760779.110805:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768343776385 time 768387232743 state 0x0 ref 1 00000100:00100000:2.0:1789760779.110829:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768387493159 time 768387232743 state 0x1 ref 1 00000100:00100000:2.0:1789760779.111170:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768387493159 time 768442098816 state 0x0 ref 1 00000100:00100000:2.0:1789760779.111186:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484336256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760779.111238:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24522752, mtime: 0x6aad950a, atime: 0x0, ctime: 0x6aad950a, blocks: 0xc007 00010000:00010000:2.0:1789760779.111246:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.111260:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:2.0:1789760779.111264:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760779.111270:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24522752->24526847] (req 24522752->24526847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.111284:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24522752->24526847] (req 24522752->24526847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760779.143639:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973305 is committed 00002000:00010000:0.0:1789760779.143716:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24522752 -> 24526848 00002000:00010000:0.0:1789760779.143721:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760778 -> 1789760779 00002000:00010000:0.0:1789760779.143724:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760778 -> 1789760779 00010000:00010000:0.0:1789760779.143736:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24522752->24526847] (req 24522752->24526847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.143748:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24522752->24526847] (req 24522752->24526847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.143757:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24522752->24526847] (req 24522752->24526847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.143765:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760779.143769:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24522752->24526847] (req 24522752->24526847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.143779:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24522752->24526847] (req 24522752->24526847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.143804:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24522752->24526847] (req 24522752->24526847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.143813:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209d72 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24522752->24526847] (req 24522752->24526847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760779.143983:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484336256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32795us (33342us total) trans 4294973305 rc 0/0 00000100:00100000:0.0:1789760779.143992:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6016 00000100:00100000:0.0:1789760779.144003:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768387493159 time 768442098816 state 0x2 ref 0 00000100:00100000:1.0:1789760779.147408:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760779.147477:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484336384 00000100:00100000:2.0:1789760779.147488:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484336384 00000100:00100000:2.0:1789760779.147540:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768387493159 time 768442098816 state 0x0 ref 1 00000100:00100000:2.0:1789760779.147555:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768442359232 time 768442098816 state 0x1 ref 1 00000100:00100000:2.0:1789760779.147587:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768442359232 time 768478518665 state 0x0 ref 1 00000100:00100000:2.0:1789760779.147599:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484336384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760779.147646:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24526848, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:2.0:1789760779.147655:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.147672:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:2.0:1789760779.147677:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760779.147688:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24526848->24530943] (req 24526848->24530943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.147702:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24526848->24530943] (req 24526848->24530943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760779.148318:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfb0f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508938496:0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760779.148374:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760779.148428:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508938496 00000100:00100000:3.0:1789760779.148434:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508938496 00000100:00100000:3.0:1789760779.148464:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 469 00000100:00100000:3.0:1789760779.148470:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abd180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508938496:12345-0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760779.148821:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508938624:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760779.148856:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760779.148923:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508938624 00000100:00100000:2.0:1789760779.148928:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508938624 00000100:00100000:2.0:1789760779.148973:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 410 00000100:00100000:2.0:1789760779.148980:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508938624:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:2.0:1789760779.148994:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527499264 pending 1703936 free 3833593856 avail 3793747968 00000100:00100000:2.0:1789760779.149049:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f2a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508938624:12345-0@lo:13:osp-pre-0-1.0 Request processed in 68us (191us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760779.149056:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 410 00000100:00080000:0.0:1789760779.149203:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760779.149238:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508938624:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760779.149348:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508938752:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760779.149376:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000004:00000024:3.0:1789760779.149425:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:3.0:1789760779.149462:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abd180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508938496:12345-0@lo:41:osp-pre-0-1.0 Request processed in 991us (1087us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760779.149469:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 469 00000100:00100000:2.0:1789760779.149493:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508938752 00000100:00100000:2.0:1789760779.149498:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508938752 00000100:00100000:2.0:1789760779.149534:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 411 00000100:00100000:2.0:1789760779.149540:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508938752:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:2.0:1789760779.149552:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527499264 pending 1703936 free 3833593856 avail 3793747968 00000100:00100000:2.0:1789760779.149580:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508938752:12345-0@lo:13:osp-pre-0-0.0 Request processed in 40us (204us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760779.149586:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 411 00010000:00010000:3.0:1789760779.149601:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760779.149625:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfb0f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508938496:0@lo:41:osp-pre-0-1.0 00000100:00080000:0.0:1789760779.149632:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760779.149652:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508938752:0@lo:13:osp-pre-0-0.0 00002000:00010000:0.0:1789760779.176268:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24526848 -> 24530944 00010000:00010000:0.0:1789760779.176288:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24526848->24530943] (req 24526848->24530943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.176302:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24526848->24530943] (req 24526848->24530943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760779.176307:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973306 is committed 00010000:00010000:0.0:1789760779.176312:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24526848->24530943] (req 24526848->24530943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.176318:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760779.176323:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24526848->24530943] (req 24526848->24530943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.176333:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24526848->24530943] (req 24526848->24530943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.176353:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24526848->24530943] (req 24526848->24530943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.176360:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209d79 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24526848->24530943] (req 24526848->24530943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760779.176488:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484336384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28885us (29083us total) trans 4294973306 rc 0/0 00000100:00100000:0.0:1789760779.176499:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6017 00000100:00100000:0.0:1789760779.176509:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768442359232 time 768478518665 state 0x2 ref 0 00000100:00100000:2.0:1789760779.180327:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760779.180442:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484336512 00000100:00100000:0.0:1789760779.180452:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484336512 00000100:00100000:0.0:1789760779.180511:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768442359232 time 768478518665 state 0x0 ref 1 00000100:00100000:0.0:1789760779.180527:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768478779081 time 768478518665 state 0x1 ref 1 00000100:00100000:0.0:1789760779.180560:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768478779081 time 768511490268 state 0x0 ref 1 00000100:00100000:0.0:1789760779.180574:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484336512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760779.180623:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24530944, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:0.0:1789760779.180631:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.180647:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:0.0:1789760779.180651:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760779.180658:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24530944->24535039] (req 24530944->24535039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.180671:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24530944->24535039] (req 24530944->24535039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760779.214625:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24530944 -> 24535040 00010000:00010000:2.0:1789760779.214645:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24530944->24535039] (req 24530944->24535039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.214662:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24530944->24535039] (req 24530944->24535039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.214671:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24530944->24535039] (req 24530944->24535039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.214678:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760779.214683:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24530944->24535039] (req 24530944->24535039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.214691:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24530944->24535039] (req 24530944->24535039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.214710:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24530944->24535039] (req 24530944->24535039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.214716:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209d80 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24530944->24535039] (req 24530944->24535039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760779.214812:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484336512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34236us (34494us total) trans 4294973307 rc 0/0 00000100:00100000:2.0:1789760779.214830:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6018 00000100:00100000:2.0:1789760779.214840:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768478779081 time 768511490268 state 0x2 ref 0 00000001:00080000:3.0:1789760779.216289:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973307 is committed 00000100:00100000:1.0:1789760779.218256:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760779.218462:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484336640 00000100:00100000:3.0:1789760779.218472:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484336640 00000100:00100000:3.0:1789760779.218527:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768478779081 time 768511490268 state 0x0 ref 1 00000100:00100000:3.0:1789760779.218544:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768511750684 time 768511490268 state 0x1 ref 1 00000100:00100000:3.0:1789760779.218576:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768511750684 time 768549507852 state 0x0 ref 1 00000100:00100000:3.0:1789760779.218589:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484336640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760779.218636:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24535040, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:3.0:1789760779.218645:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.218660:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:3.0:1789760779.218665:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760779.218672:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24535040->24539135] (req 24535040->24539135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.218685:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24535040->24539135] (req 24535040->24539135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760779.251525:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973308 is committed 00002000:00010000:3.0:1789760779.251531:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24535040 -> 24539136 00010000:00010000:3.0:1789760779.251545:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24535040->24539135] (req 24535040->24539135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.251558:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24535040->24539135] (req 24535040->24539135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.251567:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24535040->24539135] (req 24535040->24539135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.251573:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760779.251576:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24535040->24539135] (req 24535040->24539135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.251582:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24535040->24539135] (req 24535040->24539135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.251597:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24535040->24539135] (req 24535040->24539135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.251602:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209d87 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24535040->24539135] (req 24535040->24539135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760779.251677:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484336640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33086us (33428us total) trans 4294973308 rc 0/0 00000100:00100000:3.0:1789760779.251687:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6019 00000100:00100000:3.0:1789760779.251698:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768511750684 time 768549507852 state 0x2 ref 0 00000100:00100000:0.0:1789760779.254637:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760779.254980:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484336768 00000100:00100000:2.0:1789760779.254991:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484336768 00000100:00100000:2.0:1789760779.255059:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768511750684 time 768549507852 state 0x0 ref 1 00000100:00100000:2.0:1789760779.255160:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768549768268 time 768549507852 state 0x1 ref 1 00000100:00100000:2.0:1789760779.255185:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768549768268 time 768586115441 state 0x0 ref 1 00000100:00100000:2.0:1789760779.255199:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484336768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760779.255252:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24539136, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:2.0:1789760779.255262:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.255277:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:2.0:1789760779.255281:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760779.255287:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24539136->24543231] (req 24539136->24543231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.255300:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24539136->24543231] (req 24539136->24543231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760779.287589:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973309 is committed 00002000:00010000:2.0:1789760779.287860:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24539136 -> 24543232 00010000:00010000:2.0:1789760779.287876:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24539136->24543231] (req 24539136->24543231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.287888:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24539136->24543231] (req 24539136->24543231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.287900:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24539136->24543231] (req 24539136->24543231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.287907:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760779.287911:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24539136->24543231] (req 24539136->24543231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.287920:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24539136->24543231] (req 24539136->24543231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.287938:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24539136->24543231] (req 24539136->24543231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.287946:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209d8e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24539136->24543231] (req 24539136->24543231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760779.288047:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484336768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32837us (33403us total) trans 4294973309 rc 0/0 00000100:00100000:2.0:1789760779.288058:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6020 00000100:00100000:2.0:1789760779.288215:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768549768268 time 768586115441 state 0x2 ref 0 00000100:00100000:2.0:1789760779.292164:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760779.292386:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484336896 00000100:00100000:0.0:1789760779.292396:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484336896 00000100:00100000:0.0:1789760779.292461:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768549768268 time 768586115441 state 0x0 ref 1 00000100:00100000:0.0:1789760779.292477:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768586375857 time 768586115441 state 0x1 ref 1 00000100:00100000:0.0:1789760779.292509:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768586375857 time 768623439997 state 0x0 ref 1 00000100:00100000:0.0:1789760779.292526:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484336896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760779.292573:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24543232, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:0.0:1789760779.292581:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.292596:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:0.0:1789760779.292600:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760779.292606:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24543232->24547327] (req 24543232->24547327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.292620:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24543232->24547327] (req 24543232->24547327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760779.335098:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973310 is committed 00002000:00010000:0.0:1789760779.335280:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24543232 -> 24547328 00010000:00010000:0.0:1789760779.335296:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24543232->24547327] (req 24543232->24547327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.335309:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24543232->24547327] (req 24543232->24547327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.335317:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24543232->24547327] (req 24543232->24547327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.335324:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760779.335329:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24543232->24547327] (req 24543232->24547327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.335415:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24543232->24547327] (req 24543232->24547327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.335436:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24543232->24547327] (req 24543232->24547327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.335444:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209d95 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24543232->24547327] (req 24543232->24547327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760779.335543:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484336896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43016us (43383us total) trans 4294973310 rc 0/0 00000100:00100000:0.0:1789760779.335553:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6021 00000100:00100000:0.0:1789760779.335564:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768586375857 time 768623439997 state 0x2 ref 0 00000100:00100000:1.0:1789760779.339005:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760779.339116:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484337024 00000100:00100000:0.0:1789760779.339195:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484337024 00000100:00100000:0.0:1789760779.339257:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768586375857 time 768623439997 state 0x0 ref 1 00000100:00100000:0.0:1789760779.339273:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768623700413 time 768623439997 state 0x1 ref 1 00000100:00100000:0.0:1789760779.339306:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768623700413 time 768670236350 state 0x0 ref 1 00000100:00100000:0.0:1789760779.339320:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484337024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760779.339380:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24547328, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:0.0:1789760779.339389:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.339404:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:0.0:1789760779.339409:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760779.339416:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24547328->24551423] (req 24547328->24551423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.339430:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24547328->24551423] (req 24547328->24551423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760779.365305:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973311 is committed 00002000:00010000:0.0:1789760779.365474:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24547328 -> 24551424 00010000:00010000:0.0:1789760779.365490:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24547328->24551423] (req 24547328->24551423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.365503:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24547328->24551423] (req 24547328->24551423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.365512:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24547328->24551423] (req 24547328->24551423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.365520:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760779.365524:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24547328->24551423] (req 24547328->24551423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.365533:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24547328->24551423] (req 24547328->24551423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.365554:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24547328->24551423] (req 24547328->24551423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.365562:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209d9c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24547328->24551423] (req 24547328->24551423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760779.365663:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484337024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26341us (26663us total) trans 4294973311 rc 0/0 00000100:00100000:0.0:1789760779.365673:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6022 00000100:00100000:0.0:1789760779.365683:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768623700413 time 768670236350 state 0x2 ref 0 00000100:00100000:1.0:1789760779.369461:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760779.369548:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484337152 00000100:00100000:3.0:1789760779.369560:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484337152 00000100:00100000:3.0:1789760779.369617:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768623700413 time 768670236350 state 0x0 ref 1 00000100:00100000:3.0:1789760779.369635:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768670496766 time 768670236350 state 0x1 ref 1 00000100:00100000:3.0:1789760779.369660:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768670496766 time 768700590882 state 0x0 ref 1 00000100:00100000:3.0:1789760779.369673:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484337152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760779.369722:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24551424, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:3.0:1789760779.369731:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.369746:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:3.0:1789760779.369751:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760779.369756:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24551424->24555519] (req 24551424->24555519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.369770:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24551424->24555519] (req 24551424->24555519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760779.406928:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973312 is committed 00002000:00010000:3.0:1789760779.407196:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24551424 -> 24555520 00010000:00010000:3.0:1789760779.407213:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24551424->24555519] (req 24551424->24555519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.407226:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24551424->24555519] (req 24551424->24555519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.407235:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24551424->24555519] (req 24551424->24555519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.407242:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760779.407247:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24551424->24555519] (req 24551424->24555519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.407256:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24551424->24555519] (req 24551424->24555519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.407275:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24551424->24555519] (req 24551424->24555519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.407282:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209da3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24551424->24555519] (req 24551424->24555519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760779.407608:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484337152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37932us (38150us total) trans 4294973312 rc 0/0 00000100:00100000:3.0:1789760779.407622:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6023 00000100:00100000:3.0:1789760779.407631:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768670496766 time 768700590882 state 0x2 ref 0 00000100:00100000:2.0:1789760779.412640:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760779.412764:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484337280 00000100:00100000:0.0:1789760779.412774:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484337280 00000100:00100000:0.0:1789760779.412894:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768670496766 time 768700590882 state 0x0 ref 1 00000100:00100000:0.0:1789760779.412910:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768700851298 time 768700590882 state 0x1 ref 1 00000100:00100000:0.0:1789760779.413051:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768700851298 time 768743980230 state 0x0 ref 1 00000100:00100000:0.0:1789760779.413109:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484337280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760779.413172:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24555520, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:0.0:1789760779.413182:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.413198:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:0.0:1789760779.413202:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760779.413209:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24555520->24559615] (req 24555520->24559615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.413229:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24555520->24559615] (req 24555520->24559615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760779.487242:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973313 is committed 00002000:00010000:1.0:1789760779.487362:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24555520 -> 24559616 00010000:00010000:1.0:1789760779.487378:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24555520->24559615] (req 24555520->24559615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.487391:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24555520->24559615] (req 24555520->24559615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.487400:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24555520->24559615] (req 24555520->24559615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.487407:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760779.487411:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24555520->24559615] (req 24555520->24559615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.487421:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24555520->24559615] (req 24555520->24559615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.487441:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24555520->24559615] (req 24555520->24559615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.487449:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209daa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24555520->24559615] (req 24555520->24559615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760779.487547:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484337280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 74480us (74913us total) trans 4294973313 rc 0/0 00000100:00100000:1.0:1789760779.487558:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6024 00000100:00100000:1.0:1789760779.487567:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768700851298 time 768743980230 state 0x2 ref 0 00000100:00100000:1.0:1789760779.491267:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760779.491412:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484337408 00000100:00100000:0.0:1789760779.491423:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484337408 00000100:00100000:0.0:1789760779.491485:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768700851298 time 768743980230 state 0x0 ref 1 00000100:00100000:0.0:1789760779.491502:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768744240646 time 768743980230 state 0x1 ref 1 00000100:00100000:0.0:1789760779.491532:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768744240646 time 768822462856 state 0x0 ref 1 00000100:00100000:0.0:1789760779.491546:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484337408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760779.491595:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24559616, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:0.0:1789760779.491605:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.491623:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:0.0:1789760779.491627:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760779.491634:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24559616->24563711] (req 24559616->24563711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.491648:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24559616->24563711] (req 24559616->24563711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760779.538458:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973314 is committed 00002000:00010000:3.0:1789760779.539239:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24559616 -> 24563712 00010000:00010000:3.0:1789760779.539264:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24559616->24563711] (req 24559616->24563711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.539280:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24559616->24563711] (req 24559616->24563711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.539289:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24559616->24563711] (req 24559616->24563711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.539297:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760779.539302:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24559616->24563711] (req 24559616->24563711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.539312:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24559616->24563711] (req 24559616->24563711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.539337:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24559616->24563711] (req 24559616->24563711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.539344:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf3826396209db1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24559616->24563711] (req 24559616->24563711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760779.539515:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484337408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47966us (48251us total) trans 4294973314 rc 0/0 00000100:00100000:3.0:1789760779.539528:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6025 00000100:00100000:3.0:1789760779.539540:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768744240646 time 768822462856 state 0x2 ref 0 00000100:00100000:0.0:1789760779.545786:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760779.545906:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484337536 00000100:00100000:2.0:1789760779.545920:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484337536 00000100:00100000:2.0:1789760779.546000:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768744240646 time 768822462856 state 0x0 ref 1 00000100:00100000:2.0:1789760779.546020:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768822723272 time 768822462856 state 0x1 ref 1 00000100:00100000:2.0:1789760779.546179:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768822723272 time 768877107151 state 0x0 ref 1 00000100:00100000:2.0:1789760779.546196:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484337536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760779.546266:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24563712, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:2.0:1789760779.546278:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.546299:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:2.0:1789760779.546305:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760779.546314:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24563712->24567807] (req 24563712->24567807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.546332:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24563712->24567807] (req 24563712->24567807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760779.574086:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973315 is committed 00002000:00010000:0.0:1789760779.574213:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24563712 -> 24567808 00010000:00010000:0.0:1789760779.574233:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24563712->24567807] (req 24563712->24567807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.574249:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24563712->24567807] (req 24563712->24567807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.574259:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24563712->24567807] (req 24563712->24567807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.574267:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760779.574272:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24563712->24567807] (req 24563712->24567807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.574281:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24563712->24567807] (req 24563712->24567807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.574309:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24563712->24567807] (req 24563712->24567807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.574317:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209db8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24563712->24567807] (req 24563712->24567807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760779.574476:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484337536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28274us (28698us total) trans 4294973315 rc 0/0 00000100:00100000:0.0:1789760779.574490:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6026 00000100:00100000:0.0:1789760779.574502:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768822723272 time 768877107151 state 0x2 ref 0 00000100:00100000:2.0:1789760779.577800:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760779.577958:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484337664 00000100:00100000:3.0:1789760779.577973:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484337664 00000100:00100000:3.0:1789760779.578059:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768822723272 time 768877107151 state 0x0 ref 1 00000100:00100000:3.0:1789760779.578383:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768877367567 time 768877107151 state 0x1 ref 1 00000100:00100000:3.0:1789760779.578484:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768877367567 time 768909412794 state 0x0 ref 1 00000100:00100000:3.0:1789760779.578501:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484337664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760779.578566:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24567808, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:3.0:1789760779.578576:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.578597:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:3.0:1789760779.578602:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760779.578609:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24567808->24571903] (req 24567808->24571903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.578626:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24567808->24571903] (req 24567808->24571903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760779.619412:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973316 is committed 00002000:00010000:3.0:1789760779.619564:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24567808 -> 24571904 00010000:00010000:3.0:1789760779.619585:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24567808->24571903] (req 24567808->24571903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.619675:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24567808->24571903] (req 24567808->24571903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.619686:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24567808->24571903] (req 24567808->24571903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.619695:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760779.619700:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24567808->24571903] (req 24567808->24571903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.619711:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24567808->24571903] (req 24567808->24571903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.619737:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24567808->24571903] (req 24567808->24571903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.619746:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dbf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24567808->24571903] (req 24567808->24571903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760779.619976:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484337664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41473us (42183us total) trans 4294973316 rc 0/0 00000100:00100000:3.0:1789760779.623539:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6027 00000100:00100000:3.0:1789760779.623556:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768877367567 time 768909412794 state 0x2 ref 0 00000100:00100000:1.0:1789760779.623952:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760779.624499:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484337792 00000100:00100000:0.0:1789760779.624512:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484337792 00000100:00100000:1.0:1789760779.624712:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768877367567 time 768909412794 state 0x0 ref 1 00000100:00100000:1.0:1789760779.624731:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768909673210 time 768909412794 state 0x1 ref 1 00000100:00100000:1.0:1789760779.624799:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768909673210 time 768955729669 state 0x0 ref 1 00000100:00100000:1.0:1789760779.624874:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484337792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760779.624944:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24571904, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:1.0:1789760779.624955:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.624975:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=768s v=(0 0) 00010000:00010000:1.0:1789760779.624980:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760779.625201:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24571904->24575999] (req 24571904->24575999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.625221:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24571904->24575999] (req 24571904->24575999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760779.663838:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb9f0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508938880:0@lo:41:osp-pre-1-0.0 00000100:00100000:2.0:1789760779.663945:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00002000:00010000:1.0:1789760779.664776:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24571904 -> 24576000 00010000:00010000:1.0:1789760779.664798:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24571904->24575999] (req 24571904->24575999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.664813:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24571904->24575999] (req 24571904->24575999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.664823:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24571904->24575999] (req 24571904->24575999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.664831:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760779.664837:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24571904->24575999] (req 24571904->24575999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.664848:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24571904->24575999] (req 24571904->24575999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.664874:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24571904->24575999] (req 24571904->24575999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.664894:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209dc6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24571904->24575999] (req 24571904->24575999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760779.665211:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508938880 00000100:00100000:2.0:1789760779.665223:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508938880 00000100:00100000:2.0:1789760779.665286:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 470 00000100:00100000:2.0:1789760779.665298:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba77100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508938880:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:2.0:1789760779.665353:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:2.0:1789760779.665424:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba77100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508938880:12345-0@lo:41:osp-pre-1-0.0 Request processed in 127us (1479us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760779.665432:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 470 00000100:00100000:2.0:1789760779.665491:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb9f0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508938880:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760779.665719:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484337792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40903us (41763us total) trans 4294973317 rc 0/0 00000100:00100000:1.0:1789760779.665733:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6028 00000100:00100000:1.0:1789760779.665745:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768909673210 time 768955729669 state 0x2 ref 0 00000001:00080000:0.0:1789760779.665796:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973317 is committed 00000100:00100000:0.0:1789760779.669436:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760779.669541:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484337920 00000100:00100000:3.0:1789760779.669555:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484337920 00000100:00100000:3.0:1789760779.669715:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768909673210 time 768955729669 state 0x0 ref 1 00000100:00100000:3.0:1789760779.669735:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768955990085 time 768955729669 state 0x1 ref 1 00000100:00100000:3.0:1789760779.669769:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768955990085 time 769000699959 state 0x0 ref 1 00000100:00100000:3.0:1789760779.669784:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484337920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760779.669966:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24576000, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:3.0:1789760779.669978:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.670108:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:3.0:1789760779.670114:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760779.670228:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24576000->24580095] (req 24576000->24580095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.670249:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24576000->24580095] (req 24576000->24580095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760779.706574:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973318 is committed 00002000:00010000:3.0:1789760779.706707:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24576000 -> 24580096 00010000:00010000:3.0:1789760779.706729:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24576000->24580095] (req 24576000->24580095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.706745:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24576000->24580095] (req 24576000->24580095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.706757:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24576000->24580095] (req 24576000->24580095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.706765:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760779.706771:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24576000->24580095] (req 24576000->24580095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.706783:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24576000->24580095] (req 24576000->24580095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.706809:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24576000->24580095] (req 24576000->24580095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.707612:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209dcd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24576000->24580095] (req 24576000->24580095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760779.707749:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484337920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37961us (38322us total) trans 4294973318 rc 0/0 00000100:00100000:3.0:1789760779.707762:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6029 00000100:00100000:3.0:1789760779.707776:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768955990085 time 769000699959 state 0x2 ref 0 00000100:00100000:2.0:1789760779.711927:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760779.712278:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484338048 00000100:00100000:0.0:1789760779.712292:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484338048 00000100:00100000:0.0:1789760779.712381:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 768955990085 time 769000699959 state 0x0 ref 1 00000100:00100000:0.0:1789760779.712401:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769000960375 time 769000699959 state 0x1 ref 1 00000100:00100000:0.0:1789760779.712505:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769000960375 time 769043435130 state 0x0 ref 1 00000100:00100000:0.0:1789760779.712521:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484338048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760779.712592:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24580096, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:0.0:1789760779.712604:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.712625:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:0.0:1789760779.712631:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760779.712639:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24580096->24584191] (req 24580096->24584191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.712656:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24580096->24584191] (req 24580096->24584191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760779.744701:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24580096 -> 24584192 00010000:00010000:2.0:1789760779.744728:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24580096->24584191] (req 24580096->24584191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.744743:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24580096->24584191] (req 24580096->24584191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.744754:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24580096->24584191] (req 24580096->24584191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.744763:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760779.744768:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24580096->24584191] (req 24580096->24584191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.744779:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24580096->24584191] (req 24580096->24584191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.744805:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24580096->24584191] (req 24580096->24584191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.744830:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209dd4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24580096->24584191] (req 24580096->24584191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760779.744968:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484338048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32442us (33046us total) trans 4294973319 rc 0/0 00000100:00100000:2.0:1789760779.744981:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6030 00000100:00100000:2.0:1789760779.744993:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769000960375 time 769043435130 state 0x2 ref 0 00000001:00080000:1.0:1789760779.745623:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973319 is committed 00000100:00100000:1.0:1789760779.748276:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760779.748383:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484338176 00000100:00100000:3.0:1789760779.748399:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484338176 00000100:00100000:3.0:1789760779.748480:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769000960375 time 769043435130 state 0x0 ref 1 00000100:00100000:3.0:1789760779.748501:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769043695546 time 769043435130 state 0x1 ref 1 00000100:00100000:3.0:1789760779.748539:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769043695546 time 769079469957 state 0x0 ref 1 00000100:00100000:3.0:1789760779.748555:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484338176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760779.748626:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24584192, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:3.0:1789760779.748637:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.748658:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:3.0:1789760779.748663:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760779.748672:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24584192->24588287] (req 24584192->24588287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.748690:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24584192->24588287] (req 24584192->24588287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760779.785630:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973320 is committed 00002000:00010000:2.0:1789760779.785971:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24584192 -> 24588288 00010000:00010000:2.0:1789760779.785995:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24584192->24588287] (req 24584192->24588287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.786011:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24584192->24588287] (req 24584192->24588287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.786021:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24584192->24588287] (req 24584192->24588287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.786041:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760779.786047:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24584192->24588287] (req 24584192->24588287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.786058:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24584192->24588287] (req 24584192->24588287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.786236:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24584192->24588287] (req 24584192->24588287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.786245:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ddb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24584192->24588287] (req 24584192->24588287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760779.786389:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484338176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37830us (38262us total) trans 4294973320 rc 0/0 00000100:00100000:2.0:1789760779.786403:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6031 00000100:00100000:2.0:1789760779.786417:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769043695546 time 769079469957 state 0x2 ref 0 00000100:00100000:3.0:1789760779.789150:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760779.789339:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484338304 00000100:00100000:1.0:1789760779.789352:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484338304 00000100:00100000:1.0:1789760779.789428:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769043695546 time 769079469957 state 0x0 ref 1 00000100:00100000:1.0:1789760779.789448:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769079730373 time 769079469957 state 0x1 ref 1 00000100:00100000:1.0:1789760779.789483:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769079730373 time 769120413441 state 0x0 ref 1 00000100:00100000:1.0:1789760779.789497:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484338304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760779.789562:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24588288, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:1.0:1789760779.789574:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.789593:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:1.0:1789760779.789599:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760779.789607:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24588288->24592383] (req 24588288->24592383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.789623:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24588288->24592383] (req 24588288->24592383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760779.827485:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973321 is committed 00002000:00010000:1.0:1789760779.827684:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24588288 -> 24592384 00010000:00010000:1.0:1789760779.827704:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24588288->24592383] (req 24588288->24592383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.827725:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24588288->24592383] (req 24588288->24592383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.827882:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24588288->24592383] (req 24588288->24592383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.827890:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760779.827894:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24588288->24592383] (req 24588288->24592383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.827904:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24588288->24592383] (req 24588288->24592383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.827929:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24588288->24592383] (req 24588288->24592383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.827937:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24588288->24592383] (req 24588288->24592383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760779.828111:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484338304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38567us (38929us total) trans 4294973321 rc 0/0 00000100:00100000:1.0:1789760779.828128:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6032 00000100:00100000:1.0:1789760779.828141:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769079730373 time 769120413441 state 0x2 ref 0 00000100:00100000:2.0:1789760779.833640:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760779.833746:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484338432 00000100:00100000:1.0:1789760779.833761:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484338432 00000100:00100000:1.0:1789760779.833846:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769079730373 time 769120413441 state 0x0 ref 1 00000100:00100000:1.0:1789760779.833865:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769120673857 time 769120413441 state 0x1 ref 1 00000100:00100000:1.0:1789760779.833900:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769120673857 time 769164829707 state 0x0 ref 1 00000100:00100000:1.0:1789760779.833915:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484338432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760779.833977:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24592384, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:1.0:1789760779.833989:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.834010:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:1.0:1789760779.834015:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760779.834023:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24592384->24596479] (req 24592384->24596479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760779.834051:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24592384->24596479] (req 24592384->24596479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760779.871720:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973322 is committed 00002000:00010000:2.0:1789760779.871988:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24592384 -> 24596480 00010000:00010000:2.0:1789760779.872010:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24592384->24596479] (req 24592384->24596479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.872025:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24592384->24596479] (req 24592384->24596479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.872044:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24592384->24596479] (req 24592384->24596479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.872052:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760779.872057:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24592384->24596479] (req 24592384->24596479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.872092:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24592384->24596479] (req 24592384->24596479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.872118:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24592384->24596479] (req 24592384->24596479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.872127:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209de9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24592384->24596479] (req 24592384->24596479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760779.872290:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484338432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38371us (38659us total) trans 4294973322 rc 0/0 00000100:00100000:2.0:1789760779.872304:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6033 00000100:00100000:2.0:1789760779.872316:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769120673857 time 769164829707 state 0x2 ref 0 00000100:00100000:3.0:1789760779.874612:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760779.874692:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484338560 00000100:00100000:2.0:1789760779.874704:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484338560 00000100:00100000:2.0:1789760779.874885:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769120673857 time 769164829707 state 0x0 ref 1 00000100:00100000:2.0:1789760779.874907:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769165090123 time 769164829707 state 0x1 ref 1 00000100:00100000:2.0:1789760779.874950:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769165090123 time 769205880549 state 0x0 ref 1 00000100:00100000:2.0:1789760779.874963:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484338560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760779.875026:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24596480, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:2.0:1789760779.875046:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.875088:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:2.0:1789760779.875093:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760779.875100:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24596480->24600575] (req 24596480->24600575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.875117:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24596480->24600575] (req 24596480->24600575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760779.937351:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973323 is committed 00002000:00010000:0.0:1789760779.937559:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24596480 -> 24600576 00010000:00010000:0.0:1789760779.937579:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24596480->24600575] (req 24596480->24600575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.937594:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24596480->24600575] (req 24596480->24600575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.937605:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24596480->24600575] (req 24596480->24600575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.937612:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760779.937617:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24596480->24600575] (req 24596480->24600575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.937628:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24596480->24600575] (req 24596480->24600575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.937651:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24596480->24600575] (req 24596480->24600575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.937658:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209df0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24596480->24600575] (req 24596480->24600575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760779.937772:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484338560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62805us (63165us total) trans 4294973323 rc 0/0 00000100:00100000:0.0:1789760779.937783:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6034 00000100:00100000:0.0:1789760779.937795:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769165090123 time 769205880549 state 0x2 ref 0 00000100:00100000:0.0:1789760779.940532:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760779.940658:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484338688 00000100:00100000:2.0:1789760779.940671:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484338688 00000100:00100000:2.0:1789760779.940745:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769165090123 time 769205880549 state 0x0 ref 1 00000100:00100000:2.0:1789760779.940764:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769206140965 time 769205880549 state 0x1 ref 1 00000100:00100000:2.0:1789760779.940860:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769206140965 time 769271790360 state 0x0 ref 1 00000100:00100000:2.0:1789760779.940876:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484338688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760779.940956:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24600576, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:2.0:1789760779.940968:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.940988:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:2.0:1789760779.940993:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760779.940999:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24600576->24604671] (req 24600576->24604671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760779.941018:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24600576->24604671] (req 24600576->24604671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760779.975416:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973324 is committed 00002000:00010000:0.0:1789760779.975534:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24600576 -> 24604672 00010000:00010000:0.0:1789760779.975557:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24600576->24604671] (req 24600576->24604671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.975572:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24600576->24604671] (req 24600576->24604671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.975582:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24600576->24604671] (req 24600576->24604671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.975590:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760779.975597:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24600576->24604671] (req 24600576->24604671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.975609:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24600576->24604671] (req 24600576->24604671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.975636:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24600576->24604671] (req 24600576->24604671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760779.975644:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209df7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24600576->24604671] (req 24600576->24604671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760779.975809:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484338688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34929us (35285us total) trans 4294973324 rc 0/0 00000100:00100000:0.0:1789760779.975826:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6035 00000100:00100000:0.0:1789760779.975837:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769206140965 time 769271790360 state 0x2 ref 0 00000100:00100000:2.0:1789760779.981687:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760779.981896:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484338816 00000100:00100000:3.0:1789760779.981911:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484338816 00000100:00100000:3.0:1789760779.981982:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769206140965 time 769271790360 state 0x0 ref 1 00000100:00100000:3.0:1789760779.982002:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769272050776 time 769271790360 state 0x1 ref 1 00000100:00100000:3.0:1789760779.982043:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769272050776 time 769312972849 state 0x0 ref 1 00000100:00100000:3.0:1789760779.982058:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98df100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484338816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760779.982318:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24604672, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:3.0:1789760779.982330:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.982350:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:3.0:1789760779.982355:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760779.982363:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24604672->24608767] (req 24604672->24608767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760779.982379:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24604672->24608767] (req 24604672->24608767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760780.027672:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973325 is committed 00002000:00010000:1.0:1789760780.027817:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24604672 -> 24608768 00010000:00010000:1.0:1789760780.027839:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24604672->24608767] (req 24604672->24608767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.027855:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24604672->24608767] (req 24604672->24608767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.027871:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24604672->24608767] (req 24604672->24608767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.027879:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760780.027885:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24604672->24608767] (req 24604672->24608767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.027896:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24604672->24608767] (req 24604672->24608767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.027923:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24604672->24608767] (req 24604672->24608767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.027933:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209dfe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24604672->24608767] (req 24604672->24608767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760780.028117:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98df100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484338816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46010us (46393us total) trans 4294973325 rc 0/0 00000100:00100000:1.0:1789760780.028129:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6036 00000100:00100000:1.0:1789760780.028143:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769272050776 time 769312972849 state 0x2 ref 0 00000100:00100000:3.0:1789760780.035836:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760780.036251:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484338944 00000100:00100000:1.0:1789760780.036265:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484338944 00000100:00100000:1.0:1789760780.036360:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769272050776 time 769312972849 state 0x0 ref 1 00000100:00100000:1.0:1789760780.036380:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769313233265 time 769312972849 state 0x1 ref 1 00000100:00100000:1.0:1789760780.036420:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769313233265 time 769367350177 state 0x0 ref 1 00000100:00100000:1.0:1789760780.036434:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484338944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760780.036499:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24608768, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:1.0:1789760780.036510:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.036529:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:1.0:1789760780.036534:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760780.036542:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24608768->24612863] (req 24608768->24612863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.036558:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24608768->24612863] (req 24608768->24612863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1789760780.037552:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:1.0:1789760780.060241:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24608768 -> 24612864 00010000:00010000:1.0:1789760780.060271:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24608768->24612863] (req 24608768->24612863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760780.060274:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973326 is committed 00010000:00010000:1.0:1789760780.060285:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24608768->24612863] (req 24608768->24612863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.060306:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24608768->24612863] (req 24608768->24612863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.060314:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760780.060320:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24608768->24612863] (req 24608768->24612863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.060332:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24608768->24612863] (req 24608768->24612863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.060355:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24608768->24612863] (req 24608768->24612863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.060364:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf3826396209e05 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24608768->24612863] (req 24608768->24612863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760780.060495:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484338944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24057us (24667us total) trans 4294973326 rc 0/0 00000100:00100000:1.0:1789760780.060505:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6037 00000100:00100000:1.0:1789760780.060518:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769313233265 time 769367350177 state 0x2 ref 0 00000100:00100000:3.0:1789760780.063880:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760780.064011:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484339072 00000100:00100000:0.0:1789760780.064023:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484339072 00000100:00100000:0.0:1789760780.064267:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769313233265 time 769367350177 state 0x0 ref 1 00000100:00100000:0.0:1789760780.064281:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769367610593 time 769367350177 state 0x1 ref 1 00000100:00100000:0.0:1789760780.064310:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769367610593 time 769395239813 state 0x0 ref 1 00000100:00100000:0.0:1789760780.064324:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484339072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760780.064384:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24612864, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:0.0:1789760780.064392:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.064417:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:0.0:1789760780.064421:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760780.064426:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24612864->24616959] (req 24612864->24616959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.064438:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24612864->24616959] (req 24612864->24616959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760780.094400:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973327 is committed 00002000:00010000:0.0:1789760780.094539:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24612864 -> 24616960 00010000:00010000:0.0:1789760780.094559:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24612864->24616959] (req 24612864->24616959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.094575:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24612864->24616959] (req 24612864->24616959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.094584:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24612864->24616959] (req 24612864->24616959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.094593:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760780.094599:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24612864->24616959] (req 24612864->24616959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.094609:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24612864->24616959] (req 24612864->24616959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.094636:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24612864->24616959] (req 24612864->24616959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.094644:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e0c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24612864->24616959] (req 24612864->24616959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760780.094765:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484339072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30437us (30892us total) trans 4294973327 rc 0/0 00000100:00100000:0.0:1789760780.094777:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6038 00000100:00100000:0.0:1789760780.094790:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769367610593 time 769395239813 state 0x2 ref 0 00000100:00100000:2.0:1789760780.099610:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760780.099733:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484339200 00000100:00100000:0.0:1789760780.099748:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484339200 00000100:00100000:0.0:1789760780.099835:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769367610593 time 769395239813 state 0x0 ref 1 00000100:00100000:0.0:1789760780.100014:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769395500229 time 769395239813 state 0x1 ref 1 00000100:00100000:0.0:1789760780.100209:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769395500229 time 769431138526 state 0x0 ref 1 00000100:00100000:0.0:1789760780.100227:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484339200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760780.100303:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24616960, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:0.0:1789760780.100313:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.100334:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:0.0:1789760780.100339:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760780.100346:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24616960->24621055] (req 24616960->24621055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.100363:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24616960->24621055] (req 24616960->24621055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760780.145375:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973328 is committed 00002000:00010000:3.0:1789760780.145506:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24616960 -> 24621056 00010000:00010000:3.0:1789760780.145527:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24616960->24621055] (req 24616960->24621055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.145543:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24616960->24621055] (req 24616960->24621055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.145554:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24616960->24621055] (req 24616960->24621055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.145562:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.145568:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24616960->24621055] (req 24616960->24621055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.145579:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24616960->24621055] (req 24616960->24621055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.145604:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24616960->24621055] (req 24616960->24621055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.145613:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209e13 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24616960->24621055] (req 24616960->24621055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.145734:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484339200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45506us (46132us total) trans 4294973328 rc 0/0 00000100:00100000:3.0:1789760780.145747:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6039 00000100:00100000:3.0:1789760780.145759:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769395500229 time 769431138526 state 0x2 ref 0 00000100:00100000:1.0:1789760780.149202:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760780.149430:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484339328 00000100:00100000:3.0:1789760780.149444:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484339328 00000100:00100000:3.0:1789760780.149517:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769395500229 time 769431138526 state 0x0 ref 1 00000100:00100000:3.0:1789760780.149537:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769431398942 time 769431138526 state 0x1 ref 1 00000100:00100000:3.0:1789760780.149570:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769431398942 time 769480501224 state 0x0 ref 1 00000100:00100000:3.0:1789760780.149585:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484339328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760780.149648:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24621056, mtime: 0x6aad950b, atime: 0x0, ctime: 0x6aad950b, blocks: 0xc007 00010000:00010000:3.0:1789760780.149659:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.149679:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:3.0:1789760780.149685:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760780.149693:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24621056->24625151] (req 24621056->24625151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.149710:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24621056->24625151] (req 24621056->24625151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.172454:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760780.195549:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973329 is committed 00002000:00010000:3.0:1789760780.196160:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24621056 -> 24625152 00002000:00010000:3.0:1789760780.196167:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760779 -> 1789760780 00002000:00010000:3.0:1789760780.196169:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760779 -> 1789760780 00010000:00010000:3.0:1789760780.196186:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24621056->24625151] (req 24621056->24625151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.196202:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24621056->24625151] (req 24621056->24625151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.196213:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24621056->24625151] (req 24621056->24625151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.196221:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.196227:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24621056->24625151] (req 24621056->24625151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.196236:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24621056->24625151] (req 24621056->24625151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.196261:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24621056->24625151] (req 24621056->24625151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.196269:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e1a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24621056->24625151] (req 24621056->24625151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.196400:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484339328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46811us (47206us total) trans 4294973329 rc 0/0 00000100:00100000:3.0:1789760780.196414:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6040 00000100:00100000:3.0:1789760780.196427:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769431398942 time 769480501224 state 0x2 ref 0 00000100:00100000:0.0:1789760780.202117:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760780.202303:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484339456 00000100:00100000:1.0:1789760780.202318:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484339456 00000100:00100000:1.0:1789760780.202390:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769431398942 time 769480501224 state 0x0 ref 1 00000100:00100000:1.0:1789760780.202410:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769480761640 time 769480501224 state 0x1 ref 1 00000100:00100000:1.0:1789760780.202439:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769480761640 time 769533369529 state 0x0 ref 1 00000100:00100000:1.0:1789760780.202453:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484339456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760780.202577:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24625152, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:1.0:1789760780.202587:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.202605:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:1.0:1789760780.202609:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760780.202614:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24625152->24629247] (req 24625152->24629247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.202627:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24625152->24629247] (req 24625152->24629247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760780.234207:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973330 is committed 00002000:00010000:3.0:1789760780.234356:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24625152 -> 24629248 00010000:00010000:3.0:1789760780.234380:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24625152->24629247] (req 24625152->24629247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.234396:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24625152->24629247] (req 24625152->24629247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.234412:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24625152->24629247] (req 24625152->24629247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.234431:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.234437:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24625152->24629247] (req 24625152->24629247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.234448:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24625152->24629247] (req 24625152->24629247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.234474:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24625152->24629247] (req 24625152->24629247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.234482:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf3826396209e21 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24625152->24629247] (req 24625152->24629247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.234603:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484339456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32146us (32529us total) trans 4294973330 rc 0/0 00000100:00100000:3.0:1789760780.234616:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6041 00000100:00100000:3.0:1789760780.234629:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769480761640 time 769533369529 state 0x2 ref 0 00000100:00100000:2.0:1789760780.238910:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760780.240447:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484339584 00000100:00100000:0.0:1789760780.240523:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484339584 00000100:00100000:0.0:1789760780.240685:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769480761640 time 769533369529 state 0x0 ref 1 00000100:00100000:0.0:1789760780.240705:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769533629945 time 769533369529 state 0x1 ref 1 00000100:00100000:0.0:1789760780.240743:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769533629945 time 769571673215 state 0x0 ref 1 00000100:00100000:0.0:1789760780.240759:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484339584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760780.240941:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24629248, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:0.0:1789760780.240952:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.240971:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:0.0:1789760780.240976:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760780.240984:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24629248->24633343] (req 24629248->24633343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.241000:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24629248->24633343] (req 24629248->24633343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760780.288185:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24629248 -> 24633344 00010000:00010000:0.0:1789760780.288212:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24629248->24633343] (req 24629248->24633343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.288227:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24629248->24633343] (req 24629248->24633343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.288237:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24629248->24633343] (req 24629248->24633343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.288250:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760780.288256:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24629248->24633343] (req 24629248->24633343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.288267:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24629248->24633343] (req 24629248->24633343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.288291:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24629248->24633343] (req 24629248->24633343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.288300:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e28 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24629248->24633343] (req 24629248->24633343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760780.288527:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484339584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47766us (49625us total) trans 4294973331 rc 0/0 00000100:00100000:0.0:1789760780.288540:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6042 00000100:00100000:0.0:1789760780.288553:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769533629945 time 769571673215 state 0x2 ref 0 00000001:00080000:1.0:1789760780.288641:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973331 is committed 00000100:00100000:1.0:1789760780.293529:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760780.293688:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484339712 00000100:00100000:3.0:1789760780.293702:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484339712 00000100:00100000:3.0:1789760780.293785:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769533629945 time 769571673215 state 0x0 ref 1 00000100:00100000:3.0:1789760780.293804:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769571933631 time 769571673215 state 0x1 ref 1 00000100:00100000:3.0:1789760780.293837:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769571933631 time 769624768154 state 0x0 ref 1 00000100:00100000:3.0:1789760780.293852:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484339712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760780.293922:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24633344, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:3.0:1789760780.293934:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.293955:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:3.0:1789760780.293960:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760780.293967:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24633344->24637439] (req 24633344->24637439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.293985:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24633344->24637439] (req 24633344->24637439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760780.324591:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973332 is committed 00002000:00010000:3.0:1789760780.324859:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24633344 -> 24637440 00010000:00010000:3.0:1789760780.324879:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24633344->24637439] (req 24633344->24637439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.324904:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24633344->24637439] (req 24633344->24637439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.325055:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24633344->24637439] (req 24633344->24637439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.325106:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.325113:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24633344->24637439] (req 24633344->24637439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.325125:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24633344->24637439] (req 24633344->24637439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.325162:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24633344->24637439] (req 24633344->24637439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.325171:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e2f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24633344->24637439] (req 24633344->24637439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.325293:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484339712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31437us (31833us total) trans 4294973332 rc 0/0 00000100:00100000:3.0:1789760780.325306:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6043 00000100:00100000:3.0:1789760780.325318:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769571933631 time 769624768154 state 0x2 ref 0 00000100:00100000:3.0:1789760780.333329:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760780.333596:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484339840 00000100:00100000:1.0:1789760780.333609:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484339840 00000100:00100000:1.0:1789760780.333684:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769571933631 time 769624768154 state 0x0 ref 1 00000100:00100000:1.0:1789760780.333703:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769625028570 time 769624768154 state 0x1 ref 1 00000100:00100000:1.0:1789760780.333735:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769625028570 time 769664665793 state 0x0 ref 1 00000100:00100000:1.0:1789760780.333751:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484339840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760780.333819:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24637440, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:1.0:1789760780.333831:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.333858:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:1.0:1789760780.333864:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760780.333872:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24637440->24641535] (req 24637440->24641535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.333889:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24637440->24641535] (req 24637440->24641535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760780.365463:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973333 is committed 00002000:00010000:3.0:1789760780.365657:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24637440 -> 24641536 00010000:00010000:3.0:1789760780.365678:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24637440->24641535] (req 24637440->24641535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.365693:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24637440->24641535] (req 24637440->24641535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.365703:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24637440->24641535] (req 24637440->24641535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.365718:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.365724:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24637440->24641535] (req 24637440->24641535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.365743:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24637440->24641535] (req 24637440->24641535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.365769:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24637440->24641535] (req 24637440->24641535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.365777:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf3826396209e36 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24637440->24641535] (req 24637440->24641535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.365912:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484339840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32158us (32591us total) trans 4294973333 rc 0/0 00000100:00100000:3.0:1789760780.365925:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6044 00000100:00100000:3.0:1789760780.365937:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769625028570 time 769664665793 state 0x2 ref 0 00000100:00100000:2.0:1789760780.369550:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760780.369746:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484339968 00000100:00100000:1.0:1789760780.369758:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484339968 00000100:00100000:1.0:1789760780.369839:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769625028570 time 769664665793 state 0x0 ref 1 00000100:00100000:1.0:1789760780.369858:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769664926209 time 769664665793 state 0x1 ref 1 00000100:00100000:1.0:1789760780.369892:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769664926209 time 769700822531 state 0x0 ref 1 00000100:00100000:1.0:1789760780.369907:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484339968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760780.369972:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24641536, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:1.0:1789760780.369983:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.370001:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:1.0:1789760780.370011:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760780.370019:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24641536->24645631] (req 24641536->24645631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.370047:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24641536->24645631] (req 24641536->24645631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760780.398966:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24641536 -> 24645632 00010000:00010000:1.0:1789760780.398995:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24641536->24645631] (req 24641536->24645631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.399012:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24641536->24645631] (req 24641536->24645631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.399022:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24641536->24645631] (req 24641536->24645631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.399042:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760780.399049:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24641536->24645631] (req 24641536->24645631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.399060:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24641536->24645631] (req 24641536->24645631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760780.399157:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973334 is committed 00010000:00010000:1.0:1789760780.399280:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24641536->24645631] (req 24641536->24645631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.399290:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf3826396209e3d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24641536->24645631] (req 24641536->24645631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760780.399417:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484339968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29506us (29873us total) trans 4294973334 rc 0/0 00000100:00100000:1.0:1789760780.399429:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6045 00000100:00100000:1.0:1789760780.399442:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769664926209 time 769700822531 state 0x2 ref 0 00000100:00100000:1.0:1789760780.402202:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760780.402338:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484340096 00000100:00100000:0.0:1789760780.402352:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484340096 00000100:00100000:0.0:1789760780.402429:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769664926209 time 769700822531 state 0x0 ref 1 00000100:00100000:0.0:1789760780.402449:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769701082947 time 769700822531 state 0x1 ref 1 00000100:00100000:0.0:1789760780.402499:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769701082947 time 769733429574 state 0x0 ref 1 00000100:00100000:0.0:1789760780.402652:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484340096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760780.402718:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24645632, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:0.0:1789760780.402729:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.402749:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:0.0:1789760780.402754:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760780.402762:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24645632->24649727] (req 24645632->24649727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.402778:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24645632->24649727] (req 24645632->24649727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760780.429698:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973335 is committed 00002000:00010000:3.0:1789760780.429834:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24645632 -> 24649728 00010000:00010000:3.0:1789760780.429856:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24645632->24649727] (req 24645632->24649727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.429873:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24645632->24649727] (req 24645632->24649727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.429891:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24645632->24649727] (req 24645632->24649727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.429900:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.429906:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24645632->24649727] (req 24645632->24649727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.429916:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24645632->24649727] (req 24645632->24649727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.429941:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24645632->24649727] (req 24645632->24649727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.429949:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209e44 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24645632->24649727] (req 24645632->24649727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.430278:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484340096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27624us (28081us total) trans 4294973335 rc 0/0 00000100:00100000:3.0:1789760780.430292:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6046 00000100:00100000:3.0:1789760780.430305:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769701082947 time 769733429574 state 0x2 ref 0 00000100:00100000:3.0:1789760780.434674:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760780.434774:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484340224 00000100:00100000:0.0:1789760780.434789:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484340224 00000100:00100000:0.0:1789760780.435193:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769701082947 time 769733429574 state 0x0 ref 1 00000100:00100000:0.0:1789760780.435218:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769733689990 time 769733429574 state 0x1 ref 1 00000100:00100000:0.0:1789760780.435255:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769733689990 time 769766185845 state 0x0 ref 1 00000100:00100000:0.0:1789760780.435272:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484340224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760780.435342:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24649728, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:0.0:1789760780.435357:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.435378:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:0.0:1789760780.435383:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760780.435391:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24649728->24653823] (req 24649728->24653823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.435408:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24649728->24653823] (req 24649728->24653823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760780.473210:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24649728 -> 24653824 00010000:00010000:3.0:1789760780.473238:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24649728->24653823] (req 24649728->24653823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.473259:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24649728->24653823] (req 24649728->24653823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.473269:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24649728->24653823] (req 24649728->24653823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.473277:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.473282:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24649728->24653823] (req 24649728->24653823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.473294:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24649728->24653823] (req 24649728->24653823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.473329:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24649728->24653823] (req 24649728->24653823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.473341:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf3826396209e4b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24649728->24653823] (req 24649728->24653823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.473456:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484340224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38182us (38791us total) trans 4294973336 rc 0/0 00000100:00100000:3.0:1789760780.473470:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6047 00000100:00100000:3.0:1789760780.473482:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769733689990 time 769766185845 state 0x2 ref 0 00000001:00080000:2.0:1789760780.473974:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973336 is committed 00000100:00100000:2.0:1789760780.475894:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760780.476879:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484340352 00000100:00100000:3.0:1789760780.476896:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484340352 00000100:00100000:2.0:1789760780.477199:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769733689990 time 769766185845 state 0x0 ref 1 00000100:00100000:2.0:1789760780.477268:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769766446261 time 769766185845 state 0x1 ref 1 00000100:00100000:2.0:1789760780.477333:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769766446261 time 769808263210 state 0x0 ref 1 00000100:00100000:2.0:1789760780.477357:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484340352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760780.477425:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24653824, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:2.0:1789760780.477435:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.477464:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:2.0:1789760780.477470:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760780.477479:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24653824->24657919] (req 24653824->24657919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.477495:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24653824->24657919] (req 24653824->24657919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760780.512913:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973337 is committed 00002000:00010000:3.0:1789760780.513052:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24653824 -> 24657920 00010000:00010000:3.0:1789760780.513111:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24653824->24657919] (req 24653824->24657919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.513127:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24653824->24657919] (req 24653824->24657919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.513137:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24653824->24657919] (req 24653824->24657919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.513146:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.513162:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24653824->24657919] (req 24653824->24657919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.513175:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24653824->24657919] (req 24653824->24657919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.513200:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24653824->24657919] (req 24653824->24657919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.513210:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209e52 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24653824->24657919] (req 24653824->24657919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.513333:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484340352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35973us (37445us total) trans 4294973337 rc 0/0 00000100:00100000:3.0:1789760780.513347:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6048 00000100:00100000:3.0:1789760780.513359:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769766446261 time 769808263210 state 0x2 ref 0 00000100:00100000:1.0:1789760780.516734:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760780.516835:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484340480 00000100:00100000:3.0:1789760780.516849:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484340480 00000100:00100000:3.0:1789760780.516926:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769766446261 time 769808263210 state 0x0 ref 1 00000100:00100000:3.0:1789760780.516947:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769808523626 time 769808263210 state 0x1 ref 1 00000100:00100000:3.0:1789760780.516986:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769808523626 time 769847916943 state 0x0 ref 1 00000100:00100000:3.0:1789760780.517000:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484340480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760780.517100:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24657920, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:3.0:1789760780.517111:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.517142:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:3.0:1789760780.517147:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760780.517155:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24657920->24662015] (req 24657920->24662015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.517171:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24657920->24662015] (req 24657920->24662015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760780.561726:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973338 is committed 00002000:00010000:3.0:1789760780.562048:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24657920 -> 24662016 00010000:00010000:3.0:1789760780.562105:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24657920->24662015] (req 24657920->24662015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.562129:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24657920->24662015] (req 24657920->24662015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.562140:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24657920->24662015] (req 24657920->24662015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.562149:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.562155:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24657920->24662015] (req 24657920->24662015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.562167:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24657920->24662015] (req 24657920->24662015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.562195:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24657920->24662015] (req 24657920->24662015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.562204:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209e59 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24657920->24662015] (req 24657920->24662015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.562330:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484340480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45325us (45605us total) trans 4294973338 rc 0/0 00000100:00100000:3.0:1789760780.562344:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6049 00000100:00100000:3.0:1789760780.562356:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769808523626 time 769847916943 state 0x2 ref 0 00000100:00100000:2.0:1789760780.566833:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760780.566923:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484340608 00000100:00100000:3.0:1789760780.566938:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484340608 00000100:00100000:3.0:1789760780.567013:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769808523626 time 769847916943 state 0x0 ref 1 00000100:00100000:3.0:1789760780.567044:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769848177359 time 769847916943 state 0x1 ref 1 00000100:00100000:3.0:1789760780.567280:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769848177359 time 769898208073 state 0x0 ref 1 00000100:00100000:3.0:1789760780.567297:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484340608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760780.567365:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24662016, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:3.0:1789760780.567378:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.567397:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:3.0:1789760780.567403:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760780.567411:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24662016->24666111] (req 24662016->24666111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.567429:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24662016->24666111] (req 24662016->24666111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760780.603294:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24662016 -> 24666112 00010000:00010000:3.0:1789760780.603324:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24662016->24666111] (req 24662016->24666111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.603340:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24662016->24666111] (req 24662016->24666111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.603355:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24662016->24666111] (req 24662016->24666111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.603363:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.603369:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24662016->24666111] (req 24662016->24666111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.603379:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24662016->24666111] (req 24662016->24666111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.603404:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24662016->24666111] (req 24662016->24666111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.603412:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e60 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24662016->24666111] (req 24662016->24666111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.603539:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484340608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36239us (36714us total) trans 4294973339 rc 0/0 00000100:00100000:3.0:1789760780.603551:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6050 00000100:00100000:3.0:1789760780.603564:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769848177359 time 769898208073 state 0x2 ref 0 00000001:00080000:1.0:1789760780.606186:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973339 is committed 00000100:00100000:2.0:1789760780.613747:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760780.613961:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484340736 00000100:00100000:0.0:1789760780.613974:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484340736 00000100:00100000:0.0:1789760780.614104:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769848177359 time 769898208073 state 0x0 ref 1 00000100:00100000:0.0:1789760780.614124:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769898468489 time 769898208073 state 0x1 ref 1 00000100:00100000:0.0:1789760780.614158:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769898468489 time 769945088175 state 0x0 ref 1 00000100:00100000:0.0:1789760780.614175:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484340736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760780.614246:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24666112, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:0.0:1789760780.614257:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.614279:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:0.0:1789760780.614286:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760780.614294:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24666112->24670207] (req 24666112->24670207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.614312:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24666112->24670207] (req 24666112->24670207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760780.650382:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973340 is committed 00002000:00010000:3.0:1789760780.650636:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24666112 -> 24670208 00010000:00010000:3.0:1789760780.650656:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24666112->24670207] (req 24666112->24670207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.650679:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24666112->24670207] (req 24666112->24670207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.650689:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24666112->24670207] (req 24666112->24670207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.650697:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.650702:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24666112->24670207] (req 24666112->24670207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.650712:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24666112->24670207] (req 24666112->24670207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.650738:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24666112->24670207] (req 24666112->24670207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.650746:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209e67 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24666112->24670207] (req 24666112->24670207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.650938:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484340736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36761us (37201us total) trans 4294973340 rc 0/0 00000100:00100000:3.0:1789760780.650950:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6051 00000100:00100000:3.0:1789760780.650962:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769898468489 time 769945088175 state 0x2 ref 0 00000100:00100000:0.0:1789760780.657200:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760780.657345:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484340864 00000100:00100000:2.0:1789760780.657358:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484340864 00000100:00100000:2.0:1789760780.657443:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769898468489 time 769945088175 state 0x0 ref 1 00000100:00100000:2.0:1789760780.657461:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769945348591 time 769945088175 state 0x1 ref 1 00000100:00100000:2.0:1789760780.657497:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769945348591 time 769988427039 state 0x0 ref 1 00000100:00100000:2.0:1789760780.657518:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484340864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760780.657588:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24670208, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:2.0:1789760780.657598:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.657617:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=769s v=(0 0) 00010000:00010000:2.0:1789760780.657621:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760780.657628:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24670208->24674303] (req 24670208->24674303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.657643:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24670208->24674303] (req 24670208->24674303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760780.699443:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973341 is committed 00002000:00010000:3.0:1789760780.699628:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24670208 -> 24674304 00010000:00010000:3.0:1789760780.699649:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24670208->24674303] (req 24670208->24674303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.699741:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24670208->24674303] (req 24670208->24674303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.699752:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24670208->24674303] (req 24670208->24674303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.699760:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.699767:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24670208->24674303] (req 24670208->24674303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.699777:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24670208->24674303] (req 24670208->24674303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.699816:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24670208->24674303] (req 24670208->24674303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.699826:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209e6e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24670208->24674303] (req 24670208->24674303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.699947:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484340864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42427us (42757us total) trans 4294973341 rc 0/0 00000100:00100000:3.0:1789760780.699959:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6052 00000100:00100000:3.0:1789760780.699970:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769945348591 time 769988427039 state 0x2 ref 0 00000100:00100000:2.0:1789760780.706863:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760780.706970:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484340992 00000100:00100000:3.0:1789760780.706984:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484340992 00000100:00100000:3.0:1789760780.707091:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769945348591 time 769988427039 state 0x0 ref 1 00000100:00100000:3.0:1789760780.707115:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769988687455 time 769988427039 state 0x1 ref 1 00000100:00100000:3.0:1789760780.707151:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769988687455 time 770038081629 state 0x0 ref 1 00000100:00100000:3.0:1789760780.707167:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484340992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760780.707243:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24674304, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:3.0:1789760780.707254:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.707274:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:3.0:1789760780.707280:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760780.707355:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24674304->24678399] (req 24674304->24678399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.707375:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24674304->24678399] (req 24674304->24678399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760780.750298:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973342 is committed 00002000:00010000:1.0:1789760780.750434:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24674304 -> 24678400 00010000:00010000:1.0:1789760780.750453:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24674304->24678399] (req 24674304->24678399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.750468:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24674304->24678399] (req 24674304->24678399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.750480:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24674304->24678399] (req 24674304->24678399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.750488:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760780.750492:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24674304->24678399] (req 24674304->24678399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.750502:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24674304->24678399] (req 24674304->24678399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.750525:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24674304->24678399] (req 24674304->24678399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.750533:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf3826396209e75 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24674304->24678399] (req 24674304->24678399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760780.750685:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484340992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43514us (43829us total) trans 4294973342 rc 0/0 00000100:00100000:1.0:1789760780.750697:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6053 00000100:00100000:1.0:1789760780.750710:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769988687455 time 770038081629 state 0x2 ref 0 00000100:00100000:2.0:1789760780.754341:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760780.754465:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484341120 00000100:00100000:0.0:1789760780.754479:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484341120 00000100:00100000:0.0:1789760780.754562:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 769988687455 time 770038081629 state 0x0 ref 1 00000100:00100000:0.0:1789760780.754580:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770038342045 time 770038081629 state 0x1 ref 1 00000100:00100000:0.0:1789760780.754615:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770038342045 time 770085545299 state 0x0 ref 1 00000100:00100000:0.0:1789760780.754630:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484341120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760780.754696:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24678400, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:0.0:1789760780.754707:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.754727:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:0.0:1789760780.754733:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760780.754741:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24678400->24682495] (req 24678400->24682495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.754758:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24678400->24682495] (req 24678400->24682495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760780.798981:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973343 is committed 00002000:00010000:0.0:1789760780.799181:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24678400 -> 24682496 00010000:00010000:0.0:1789760780.799202:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24678400->24682495] (req 24678400->24682495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.799218:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24678400->24682495] (req 24678400->24682495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.799228:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24678400->24682495] (req 24678400->24682495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.799236:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760780.799242:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24678400->24682495] (req 24678400->24682495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.799253:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24678400->24682495] (req 24678400->24682495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.799279:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24678400->24682495] (req 24678400->24682495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.799289:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e7c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24678400->24682495] (req 24678400->24682495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760780.799407:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484341120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44772us (45074us total) trans 4294973343 rc 0/0 00000100:00100000:0.0:1789760780.799423:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6054 00000100:00100000:0.0:1789760780.799436:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770038342045 time 770085545299 state 0x2 ref 0 00000100:00100000:1.0:1789760780.804526:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760780.805559:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484341248 00000100:00100000:0.0:1789760780.805576:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484341248 00000100:00100000:0.0:1789760780.805716:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770038342045 time 770085545299 state 0x0 ref 1 00000100:00100000:0.0:1789760780.805739:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770085805715 time 770085545299 state 0x1 ref 1 00000100:00100000:0.0:1789760780.805822:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770085805715 time 770136752192 state 0x0 ref 1 00000100:00100000:0.0:1789760780.805840:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484341248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760780.805904:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24682496, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:0.0:1789760780.805916:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.805936:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:0.0:1789760780.805941:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760780.805948:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24682496->24686591] (req 24682496->24686591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.805966:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24682496->24686591] (req 24682496->24686591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760780.847613:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24682496 -> 24686592 00010000:00010000:3.0:1789760780.847641:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24682496->24686591] (req 24682496->24686591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.847663:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24682496->24686591] (req 24682496->24686591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.847673:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24682496->24686591] (req 24682496->24686591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.847682:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.847687:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24682496->24686591] (req 24682496->24686591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.847696:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24682496->24686591] (req 24682496->24686591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.847721:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24682496->24686591] (req 24682496->24686591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.847729:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf3826396209e83 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24682496->24686591] (req 24682496->24686591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.847860:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484341248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42019us (43341us total) trans 4294973344 rc 0/0 00000100:00100000:3.0:1789760780.847873:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6055 00000100:00100000:3.0:1789760780.847886:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770085805715 time 770136752192 state 0x2 ref 0 00000001:00080000:3.0:1789760780.847940:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973344 is committed 00000100:00100000:3.0:1789760780.851642:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760780.851784:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484341376 00000100:00100000:1.0:1789760780.851796:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484341376 00000100:00100000:1.0:1789760780.851865:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770085805715 time 770136752192 state 0x0 ref 1 00000100:00100000:1.0:1789760780.851884:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770137012608 time 770136752192 state 0x1 ref 1 00000100:00100000:1.0:1789760780.851955:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770137012608 time 770182885319 state 0x0 ref 1 00000100:00100000:1.0:1789760780.851970:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484341376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760780.852042:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24686592, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:1.0:1789760780.852053:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.852116:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:1.0:1789760780.852121:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760780.852128:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24686592->24690687] (req 24686592->24690687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.852145:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24686592->24690687] (req 24686592->24690687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760780.882970:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973345 is committed 00002000:00010000:1.0:1789760780.883239:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24686592 -> 24690688 00010000:00010000:1.0:1789760780.883273:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24686592->24690687] (req 24686592->24690687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.883287:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24686592->24690687] (req 24686592->24690687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.883297:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24686592->24690687] (req 24686592->24690687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.883306:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760780.883312:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24686592->24690687] (req 24686592->24690687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.883322:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24686592->24690687] (req 24686592->24690687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.883350:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24686592->24690687] (req 24686592->24690687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.883359:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209e8a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24686592->24690687] (req 24686592->24690687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760780.883490:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484341376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31515us (31853us total) trans 4294973345 rc 0/0 00000100:00100000:1.0:1789760780.883502:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6056 00000100:00100000:1.0:1789760780.883514:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770137012608 time 770182885319 state 0x2 ref 0 00000100:00100000:2.0:1789760780.887593:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760780.887685:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484341504 00000100:00100000:1.0:1789760780.887700:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484341504 00000100:00100000:1.0:1789760780.887774:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770137012608 time 770182885319 state 0x0 ref 1 00000100:00100000:1.0:1789760780.887793:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770183145735 time 770182885319 state 0x1 ref 1 00000100:00100000:1.0:1789760780.887831:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770183145735 time 770218762144 state 0x0 ref 1 00000100:00100000:1.0:1789760780.887846:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484341504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760780.887914:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24690688, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:1.0:1789760780.887926:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.887947:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:1.0:1789760780.887952:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760780.887960:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24690688->24694783] (req 24690688->24694783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760780.887981:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24690688->24694783] (req 24690688->24694783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760780.928493:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973346 is committed 00002000:00010000:2.0:1789760780.928741:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24690688 -> 24694784 00010000:00010000:2.0:1789760780.928761:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24690688->24694783] (req 24690688->24694783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.928776:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24690688->24694783] (req 24690688->24694783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.928785:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24690688->24694783] (req 24690688->24694783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.928793:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760780.928799:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24690688->24694783] (req 24690688->24694783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.928809:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24690688->24694783] (req 24690688->24694783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.928835:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24690688->24694783] (req 24690688->24694783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.928844:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf3826396209e91 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24690688->24694783] (req 24690688->24694783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760780.928966:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484341504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41116us (41380us total) trans 4294973346 rc 0/0 00000100:00100000:2.0:1789760780.928978:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6057 00000100:00100000:2.0:1789760780.928990:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770183145735 time 770218762144 state 0x2 ref 0 00000100:00100000:0.0:1789760780.932453:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760780.932573:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484341632 00000100:00100000:2.0:1789760780.932585:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484341632 00000100:00100000:2.0:1789760780.932730:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770183145735 time 770218762144 state 0x0 ref 1 00000100:00100000:2.0:1789760780.932749:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770219022560 time 770218762144 state 0x1 ref 1 00000100:00100000:2.0:1789760780.932784:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770219022560 time 770263714466 state 0x0 ref 1 00000100:00100000:2.0:1789760780.932799:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484341632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760780.932934:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24694784, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:2.0:1789760780.932946:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.932967:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:2.0:1789760780.932973:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760780.932980:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24694784->24698879] (req 24694784->24698879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760780.933004:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24694784->24698879] (req 24694784->24698879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760780.966683:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973347 is committed 00002000:00010000:3.0:1789760780.966779:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24694784 -> 24698880 00010000:00010000:3.0:1789760780.966799:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24694784->24698879] (req 24694784->24698879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.966877:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24694784->24698879] (req 24694784->24698879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.966887:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24694784->24698879] (req 24694784->24698879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.966896:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760780.966902:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24694784->24698879] (req 24694784->24698879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.966913:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24694784->24698879] (req 24694784->24698879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.966938:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24694784->24698879] (req 24694784->24698879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760780.966947:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209e98 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24694784->24698879] (req 24694784->24698879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760780.967124:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484341632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34320us (34681us total) trans 4294973347 rc 0/0 00000100:00100000:3.0:1789760780.967140:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6058 00000100:00100000:3.0:1789760780.967152:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770219022560 time 770263714466 state 0x2 ref 0 00000100:00100000:3.0:1789760780.972757:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760780.972892:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484341760 00000100:00100000:0.0:1789760780.972904:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484341760 00000100:00100000:0.0:1789760780.972980:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770219022560 time 770263714466 state 0x0 ref 1 00000100:00100000:0.0:1789760780.972998:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770263974882 time 770263714466 state 0x1 ref 1 00000100:00100000:0.0:1789760780.973036:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770263974882 time 770303967317 state 0x0 ref 1 00000100:00100000:0.0:1789760780.973049:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484341760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760780.973142:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24698880, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:0.0:1789760780.973152:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.973173:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:0.0:1789760780.973178:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760780.973186:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24698880->24702975] (req 24698880->24702975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760780.973203:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24698880->24702975] (req 24698880->24702975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760781.002830:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973348 is committed 00002000:00010000:3.0:1789760781.002916:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24698880 -> 24702976 00010000:00010000:3.0:1789760781.002937:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24698880->24702975] (req 24698880->24702975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.002953:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24698880->24702975] (req 24698880->24702975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.002963:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24698880->24702975] (req 24698880->24702975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.003518:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.003527:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24698880->24702975] (req 24698880->24702975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.003539:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24698880->24702975] (req 24698880->24702975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.003565:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24698880->24702975] (req 24698880->24702975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.003574:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf3826396209e9f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24698880->24702975] (req 24698880->24702975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760781.003752:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484341760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30696us (30999us total) trans 4294973348 rc 0/0 00000100:00100000:3.0:1789760781.003765:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6059 00000100:00100000:3.0:1789760781.003778:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770263974882 time 770303967317 state 0x2 ref 0 00000100:00100000:2.0:1789760781.006492:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760781.006565:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484341888 00000100:00100000:1.0:1789760781.006579:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484341888 00000100:00100000:1.0:1789760781.006650:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770263974882 time 770303967317 state 0x0 ref 1 00000100:00100000:1.0:1789760781.006669:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770304227733 time 770303967317 state 0x1 ref 1 00000100:00100000:1.0:1789760781.006702:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770304227733 time 770337633836 state 0x0 ref 1 00000100:00100000:1.0:1789760781.006719:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484341888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760781.006853:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24702976, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:1.0:1789760781.006865:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.006885:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:1.0:1789760781.006890:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760781.006898:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24702976->24707071] (req 24702976->24707071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.006914:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24702976->24707071] (req 24702976->24707071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760781.050919:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973349 is committed 00002000:00010000:1.0:1789760781.051138:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24702976 -> 24707072 00010000:00010000:1.0:1789760781.051159:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24702976->24707071] (req 24702976->24707071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.051173:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24702976->24707071] (req 24702976->24707071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.051184:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24702976->24707071] (req 24702976->24707071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.051193:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760781.051199:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24702976->24707071] (req 24702976->24707071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.051210:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24702976->24707071] (req 24702976->24707071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.051236:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24702976->24707071] (req 24702976->24707071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.051244:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf3826396209ea6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24702976->24707071] (req 24702976->24707071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760781.051376:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484341888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44654us (44890us total) trans 4294973349 rc 0/0 00000100:00100000:1.0:1789760781.051393:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6060 00000100:00100000:1.0:1789760781.051405:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770304227733 time 770337633836 state 0x2 ref 0 00000100:00100000:0.0:1789760781.056339:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760781.056447:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484342016 00000100:00100000:2.0:1789760781.056461:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484342016 00000100:00100000:2.0:1789760781.056543:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770304227733 time 770337633836 state 0x0 ref 1 00000100:00100000:2.0:1789760781.056563:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770337894252 time 770337633836 state 0x1 ref 1 00000100:00100000:2.0:1789760781.056616:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770337894252 time 770387546382 state 0x0 ref 1 00000100:00100000:2.0:1789760781.056631:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484342016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760781.056712:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24707072, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:2.0:1789760781.056724:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.056745:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:2.0:1789760781.056750:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760781.056758:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24707072->24711167] (req 24707072->24711167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.056775:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24707072->24711167] (req 24707072->24711167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760781.086665:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973350 is committed 00002000:00010000:3.0:1789760781.086959:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24707072 -> 24711168 00010000:00010000:3.0:1789760781.086980:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24707072->24711167] (req 24707072->24711167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.086996:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24707072->24711167] (req 24707072->24711167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.087006:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24707072->24711167] (req 24707072->24711167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.087015:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.087029:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24707072->24711167] (req 24707072->24711167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.087040:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24707072->24711167] (req 24707072->24711167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.087086:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24707072->24711167] (req 24707072->24711167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.087102:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209ead lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24707072->24711167] (req 24707072->24711167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760781.087311:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484342016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30676us (30980us total) trans 4294973350 rc 0/0 00000100:00100000:3.0:1789760781.087324:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6061 00000100:00100000:3.0:1789760781.087336:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770337894252 time 770387546382 state 0x2 ref 0 00000100:00100000:3.0:1789760781.091763:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760781.092879:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484342144 00000100:00100000:2.0:1789760781.092897:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484342144 00000100:00100000:2.0:1789760781.092975:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770337894252 time 770387546382 state 0x0 ref 1 00000100:00100000:2.0:1789760781.092996:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770387806798 time 770387546382 state 0x1 ref 1 00000100:00100000:2.0:1789760781.093103:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770387806798 time 770424033074 state 0x0 ref 1 00000100:00100000:2.0:1789760781.093120:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484342144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760781.093201:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24711168, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:2.0:1789760781.093212:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.093233:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:2.0:1789760781.093238:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760781.093246:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24711168->24715263] (req 24711168->24715263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.093264:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24711168->24715263] (req 24711168->24715263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760781.142684:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973351 is committed 00002000:00010000:0.0:1789760781.144920:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24711168 -> 24715264 00010000:00010000:0.0:1789760781.144948:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24711168->24715263] (req 24711168->24715263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.144971:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24711168->24715263] (req 24711168->24715263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.144981:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24711168->24715263] (req 24711168->24715263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.144998:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760781.145004:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24711168->24715263] (req 24711168->24715263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.145014:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24711168->24715263] (req 24711168->24715263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.145168:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24711168->24715263] (req 24711168->24715263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.145178:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209eb4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24711168->24715263] (req 24711168->24715263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760781.145324:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484342144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52200us (53568us total) trans 4294973351 rc 0/0 00000100:00100000:0.0:1789760781.145336:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6062 00000100:00100000:0.0:1789760781.145348:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770387806798 time 770424033074 state 0x2 ref 0 00000100:00100000:2.0:1789760781.148887:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760781.149056:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484342272 00000100:00100000:3.0:1789760781.149102:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484342272 00000100:00100000:3.0:1789760781.149181:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770387806798 time 770424033074 state 0x0 ref 1 00000100:00100000:3.0:1789760781.149201:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770424293490 time 770424033074 state 0x1 ref 1 00000100:00100000:3.0:1789760781.149229:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770424293490 time 770480160208 state 0x0 ref 1 00000100:00100000:3.0:1789760781.149243:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484342272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760781.149311:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24715264, mtime: 0x6aad950c, atime: 0x0, ctime: 0x6aad950c, blocks: 0xc007 00010000:00010000:3.0:1789760781.149323:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.149343:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:3.0:1789760781.149348:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760781.149356:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24715264->24719359] (req 24715264->24719359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.149373:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24715264->24719359] (req 24715264->24719359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760781.189209:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973352 is committed 00002000:00010000:0.0:1789760781.189335:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24715264 -> 24719360 00002000:00010000:0.0:1789760781.189342:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760780 -> 1789760781 00002000:00010000:0.0:1789760781.189345:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760780 -> 1789760781 00010000:00010000:0.0:1789760781.189481:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24715264->24719359] (req 24715264->24719359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.189496:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24715264->24719359] (req 24715264->24719359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.189505:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24715264->24719359] (req 24715264->24719359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.189512:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760781.189517:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24715264->24719359] (req 24715264->24719359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.189527:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24715264->24719359] (req 24715264->24719359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.189553:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24715264->24719359] (req 24715264->24719359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.189562:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf3826396209ebb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24715264->24719359] (req 24715264->24719359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760781.189687:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484342272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40439us (40808us total) trans 4294973352 rc 0/0 00000100:00100000:0.0:1789760781.189700:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6063 00000100:00100000:0.0:1789760781.189714:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770424293490 time 770480160208 state 0x2 ref 0 00000100:00100000:2.0:1789760781.193263:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760781.193389:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484342400 00000100:00100000:3.0:1789760781.193401:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484342400 00000100:00100000:3.0:1789760781.193465:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770424293490 time 770480160208 state 0x0 ref 1 00000100:00100000:3.0:1789760781.193479:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770480420624 time 770480160208 state 0x1 ref 1 00000100:00100000:3.0:1789760781.193507:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770480420624 time 770524437497 state 0x0 ref 1 00000100:00100000:3.0:1789760781.193521:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484342400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760781.193588:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24719360, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:3.0:1789760781.193598:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.193618:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:3.0:1789760781.193623:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760781.193630:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24719360->24723455] (req 24719360->24723455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.193646:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24719360->24723455] (req 24719360->24723455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.203553:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760781.239777:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973353 is committed 00002000:00010000:3.0:1789760781.240092:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24719360 -> 24723456 00010000:00010000:3.0:1789760781.240112:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24719360->24723455] (req 24719360->24723455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.240128:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24719360->24723455] (req 24719360->24723455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.240138:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24719360->24723455] (req 24719360->24723455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.240146:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.240152:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24719360->24723455] (req 24719360->24723455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.240161:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24719360->24723455] (req 24719360->24723455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.240187:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24719360->24723455] (req 24719360->24723455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.240196:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ec2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24719360->24723455] (req 24719360->24723455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760781.240315:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484342400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46789us (47061us total) trans 4294973353 rc 0/0 00000100:00100000:3.0:1789760781.240327:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6064 00000100:00100000:3.0:1789760781.240339:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770480420624 time 770524437497 state 0x2 ref 0 00000100:00100000:3.0:1789760781.243437:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760781.243551:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484342528 00000100:00100000:1.0:1789760781.243565:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484342528 00000100:00100000:1.0:1789760781.243642:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770480420624 time 770524437497 state 0x0 ref 1 00000100:00100000:1.0:1789760781.243660:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770524697913 time 770524437497 state 0x1 ref 1 00000100:00100000:1.0:1789760781.243697:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770524697913 time 770574626894 state 0x0 ref 1 00000100:00100000:1.0:1789760781.243711:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484342528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760781.243779:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24723456, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:1.0:1789760781.243794:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.243814:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:1.0:1789760781.243820:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760781.243827:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24723456->24727551] (req 24723456->24727551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.243844:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24723456->24727551] (req 24723456->24727551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760781.289663:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973354 is committed 00002000:00010000:3.0:1789760781.290814:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24723456 -> 24727552 00010000:00010000:3.0:1789760781.290836:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24723456->24727551] (req 24723456->24727551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.290851:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24723456->24727551] (req 24723456->24727551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.290860:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24723456->24727551] (req 24723456->24727551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.290868:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.290874:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24723456->24727551] (req 24723456->24727551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.290884:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24723456->24727551] (req 24723456->24727551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.290966:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24723456->24727551] (req 24723456->24727551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.290976:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf3826396209ec9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24723456->24727551] (req 24723456->24727551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760781.291468:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484342528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47750us (48128us total) trans 4294973354 rc 0/0 00000100:00100000:2.0:1789760781.291482:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6065 00000100:00100000:2.0:1789760781.291495:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770524697913 time 770574626894 state 0x2 ref 0 00000100:00100000:2.0:1789760781.293867:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760781.293997:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484342656 00000100:00100000:0.0:1789760781.294010:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484342656 00000100:00100000:0.0:1789760781.294282:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770524697913 time 770574626894 state 0x0 ref 1 00000100:00100000:0.0:1789760781.294301:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770574887310 time 770574626894 state 0x1 ref 1 00000100:00100000:0.0:1789760781.294370:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770574887310 time 770625299343 state 0x0 ref 1 00000100:00100000:0.0:1789760781.294387:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484342656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760781.294456:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24727552, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:0.0:1789760781.294466:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.294486:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:0.0:1789760781.294490:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760781.294498:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24727552->24731647] (req 24727552->24731647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.294514:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24727552->24731647] (req 24727552->24731647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760781.324162:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24727552 -> 24731648 00010000:00010000:0.0:1789760781.324190:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24727552->24731647] (req 24727552->24731647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.324211:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24727552->24731647] (req 24727552->24731647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.324222:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24727552->24731647] (req 24727552->24731647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.324230:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760781.324236:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24727552->24731647] (req 24727552->24731647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.324246:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24727552->24731647] (req 24727552->24731647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.324271:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24727552->24731647] (req 24727552->24731647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.324280:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209ed0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24727552->24731647] (req 24727552->24731647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760781.324399:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484342656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30009us (30535us total) trans 4294973355 rc 0/0 00000100:00100000:0.0:1789760781.324412:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6066 00000100:00100000:0.0:1789760781.324423:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770574887310 time 770625299343 state 0x2 ref 0 00000001:00080000:3.0:1789760781.325170:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973355 is committed 00000100:00100000:0.0:1789760781.328612:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760781.332614:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484342784 00000100:00100000:3.0:1789760781.332637:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484342784 00000100:00100000:2.0:1789760781.333340:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770574887310 time 770625299343 state 0x0 ref 1 00000100:00100000:2.0:1789760781.333364:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770625559759 time 770625299343 state 0x1 ref 1 00000100:00100000:2.0:1789760781.333401:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770625559759 time 770664331579 state 0x0 ref 1 00000100:00100000:2.0:1789760781.333419:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7068450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484342784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760781.333488:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24731648, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:2.0:1789760781.333499:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.333517:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:2.0:1789760781.333521:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760781.333529:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24731648->24735743] (req 24731648->24735743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.333544:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24731648->24735743] (req 24731648->24735743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760781.365841:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973356 is committed 00002000:00010000:3.0:1789760781.365925:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24731648 -> 24735744 00010000:00010000:3.0:1789760781.365944:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24731648->24735743] (req 24731648->24735743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.365959:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24731648->24735743] (req 24731648->24735743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.365969:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24731648->24735743] (req 24731648->24735743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.365977:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.365982:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24731648->24735743] (req 24731648->24735743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.365993:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24731648->24735743] (req 24731648->24735743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.366029:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24731648->24735743] (req 24731648->24735743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.366036:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209ed7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24731648->24735743] (req 24731648->24735743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760781.366195:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7068450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484342784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32774us (37592us total) trans 4294973356 rc 0/0 00000100:00100000:3.0:1789760781.366208:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6067 00000100:00100000:3.0:1789760781.366223:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770625559759 time 770664331579 state 0x2 ref 0 00000100:00100000:2.0:1789760781.371003:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760781.371197:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484342912 00000100:00100000:3.0:1789760781.371209:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484342912 00000100:00100000:3.0:1789760781.371351:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770625559759 time 770664331579 state 0x0 ref 1 00000100:00100000:3.0:1789760781.371370:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770664591995 time 770664331579 state 0x1 ref 1 00000100:00100000:3.0:1789760781.371398:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770664591995 time 770702328548 state 0x0 ref 1 00000100:00100000:3.0:1789760781.371414:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484342912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760781.371482:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24735744, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:3.0:1789760781.371490:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.371507:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:3.0:1789760781.371510:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760781.371516:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24735744->24739839] (req 24735744->24739839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.371529:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24735744->24739839] (req 24735744->24739839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760781.404668:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24735744 -> 24739840 00010000:00010000:3.0:1789760781.404694:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24735744->24739839] (req 24735744->24739839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.404713:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24735744->24739839] (req 24735744->24739839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.404722:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24735744->24739839] (req 24735744->24739839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.404728:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.404743:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24735744->24739839] (req 24735744->24739839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.404751:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24735744->24739839] (req 24735744->24739839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.404773:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24735744->24739839] (req 24735744->24739839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.404781:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209ede lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24735744->24739839] (req 24735744->24739839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760781.404886:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973357 is committed 00000100:00100000:3.0:1789760781.405042:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484342912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33624us (34045us total) trans 4294973357 rc 0/0 00000100:00100000:3.0:1789760781.405054:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6068 00000100:00100000:3.0:1789760781.405103:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770664591995 time 770702328548 state 0x2 ref 0 00000100:00100000:2.0:1789760781.408569:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760781.408637:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484343040 00000100:00100000:3.0:1789760781.408651:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484343040 00000100:00100000:3.0:1789760781.408731:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770664591995 time 770702328548 state 0x0 ref 1 00000100:00100000:3.0:1789760781.408750:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770702588964 time 770702328548 state 0x1 ref 1 00000100:00100000:3.0:1789760781.408839:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770702588964 time 770739767626 state 0x0 ref 1 00000100:00100000:3.0:1789760781.408854:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484343040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760781.408918:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24739840, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:3.0:1789760781.408929:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.408948:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:3.0:1789760781.408954:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760781.408961:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24739840->24743935] (req 24739840->24743935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.408978:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24739840->24743935] (req 24739840->24743935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760781.438238:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973358 is committed 00002000:00010000:2.0:1789760781.438332:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24739840 -> 24743936 00010000:00010000:2.0:1789760781.438353:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24739840->24743935] (req 24739840->24743935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.438369:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24739840->24743935] (req 24739840->24743935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.438380:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24739840->24743935] (req 24739840->24743935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.438388:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760781.438394:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24739840->24743935] (req 24739840->24743935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.438405:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24739840->24743935] (req 24739840->24743935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.438433:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24739840->24743935] (req 24739840->24743935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.438441:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf3826396209ee5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24739840->24743935] (req 24739840->24743935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760781.438617:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484343040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29758us (30056us total) trans 4294973358 rc 0/0 00000100:00100000:2.0:1789760781.438630:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6069 00000100:00100000:2.0:1789760781.438643:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770702588964 time 770739767626 state 0x2 ref 0 00000100:00100000:2.0:1789760781.441433:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760781.441644:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484343168 00000100:00100000:0.0:1789760781.441657:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484343168 00000100:00100000:0.0:1789760781.441736:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770702588964 time 770739767626 state 0x0 ref 1 00000100:00100000:0.0:1789760781.441755:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770740028042 time 770739767626 state 0x1 ref 1 00000100:00100000:0.0:1789760781.441961:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770740028042 time 770772889892 state 0x0 ref 1 00000100:00100000:0.0:1789760781.441989:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484343168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760781.442102:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24743936, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:0.0:1789760781.442113:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.442141:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:0.0:1789760781.442146:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760781.442154:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24743936->24748031] (req 24743936->24748031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.442171:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24743936->24748031] (req 24743936->24748031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760781.477540:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973359 is committed 00002000:00010000:1.0:1789760781.478348:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24743936 -> 24748032 00010000:00010000:1.0:1789760781.478373:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24743936->24748031] (req 24743936->24748031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.478390:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24743936->24748031] (req 24743936->24748031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.478401:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24743936->24748031] (req 24743936->24748031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.478409:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760781.478415:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24743936->24748031] (req 24743936->24748031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.478428:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24743936->24748031] (req 24743936->24748031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.478455:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24743936->24748031] (req 24743936->24748031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.478464:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf3826396209eec lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24743936->24748031] (req 24743936->24748031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760781.478613:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484343168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36621us (37186us total) trans 4294973359 rc 0/0 00000100:00100000:1.0:1789760781.478630:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6070 00000100:00100000:1.0:1789760781.478642:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770740028042 time 770772889892 state 0x2 ref 0 00000100:00100000:3.0:1789760781.484930:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760781.485047:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484343296 00000100:00100000:2.0:1789760781.485060:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484343296 00000100:00100000:2.0:1789760781.485363:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770740028042 time 770772889892 state 0x0 ref 1 00000100:00100000:2.0:1789760781.485383:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770773150308 time 770772889892 state 0x1 ref 1 00000100:00100000:2.0:1789760781.485421:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770773150308 time 770816351813 state 0x0 ref 1 00000100:00100000:2.0:1789760781.485436:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484343296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760781.485503:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24748032, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:2.0:1789760781.485514:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.485535:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:2.0:1789760781.485540:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760781.485547:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24748032->24752127] (req 24748032->24752127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.485563:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24748032->24752127] (req 24748032->24752127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760781.541294:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973360 is committed 00002000:00010000:2.0:1789760781.541646:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24748032 -> 24752128 00010000:00010000:2.0:1789760781.541666:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24748032->24752127] (req 24748032->24752127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.541684:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24748032->24752127] (req 24748032->24752127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.541695:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24748032->24752127] (req 24748032->24752127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.541703:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760781.541708:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24748032->24752127] (req 24748032->24752127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.541719:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24748032->24752127] (req 24748032->24752127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.541742:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24748032->24752127] (req 24748032->24752127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.541751:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209ef3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24748032->24752127] (req 24748032->24752127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760781.541974:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484343296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 56533us (57050us total) trans 4294973360 rc 0/0 00000100:00100000:2.0:1789760781.541989:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6071 00000100:00100000:2.0:1789760781.542003:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770773150308 time 770816351813 state 0x2 ref 0 00000100:00100000:2.0:1789760781.545918:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760781.546395:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484343424 00000100:00100000:3.0:1789760781.546410:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484343424 00000100:00100000:3.0:1789760781.546492:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770773150308 time 770816351813 state 0x0 ref 1 00000100:00100000:3.0:1789760781.546511:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770816612229 time 770816351813 state 0x1 ref 1 00000100:00100000:3.0:1789760781.546548:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770816612229 time 770877478823 state 0x0 ref 1 00000100:00100000:3.0:1789760781.546564:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484343424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760781.546634:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24752128, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:3.0:1789760781.546645:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.546664:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:3.0:1789760781.546670:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760781.546678:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24752128->24756223] (req 24752128->24756223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.546695:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24752128->24756223] (req 24752128->24756223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760781.579771:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973361 is committed 00002000:00010000:3.0:1789760781.579832:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24752128 -> 24756224 00010000:00010000:3.0:1789760781.579853:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24752128->24756223] (req 24752128->24756223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.579868:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24752128->24756223] (req 24752128->24756223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.579878:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24752128->24756223] (req 24752128->24756223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.579887:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.579893:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24752128->24756223] (req 24752128->24756223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.579904:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24752128->24756223] (req 24752128->24756223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.579931:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24752128->24756223] (req 24752128->24756223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.579940:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209efa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24752128->24756223] (req 24752128->24756223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760781.580233:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484343424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33664us (34321us total) trans 4294973361 rc 0/0 00000100:00100000:3.0:1789760781.580247:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6072 00000100:00100000:3.0:1789760781.580260:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770816612229 time 770877478823 state 0x2 ref 0 00000100:00100000:0.0:1789760781.584616:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760781.585546:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484343552 00000100:00100000:3.0:1789760781.585565:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484343552 00000100:00100000:2.0:1789760781.585736:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770816612229 time 770877478823 state 0x0 ref 1 00000100:00100000:2.0:1789760781.585757:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770877739239 time 770877478823 state 0x1 ref 1 00000100:00100000:2.0:1789760781.585800:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770877739239 time 770916730305 state 0x0 ref 1 00000100:00100000:2.0:1789760781.585815:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484343552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760781.585881:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24756224, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:2.0:1789760781.585893:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.585914:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:2.0:1789760781.585919:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760781.585928:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24756224->24760319] (req 24756224->24760319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.585945:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24756224->24760319] (req 24756224->24760319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760781.628148:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973362 is committed 00002000:00010000:2.0:1789760781.628268:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24756224 -> 24760320 00010000:00010000:2.0:1789760781.628292:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24756224->24760319] (req 24756224->24760319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.628310:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24756224->24760319] (req 24756224->24760319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.628319:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24756224->24760319] (req 24756224->24760319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.628329:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760781.628335:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24756224->24760319] (req 24756224->24760319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.628346:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24756224->24760319] (req 24756224->24760319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.628372:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24756224->24760319] (req 24756224->24760319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.628380:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f01 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24756224->24760319] (req 24756224->24760319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760781.628504:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484343552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42685us (43896us total) trans 4294973362 rc 0/0 00000100:00100000:2.0:1789760781.628515:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6073 00000100:00100000:2.0:1789760781.628528:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770877739239 time 770916730305 state 0x2 ref 0 00000100:00100000:2.0:1789760781.631377:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760781.631481:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484343680 00000100:00100000:1.0:1789760781.631494:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484343680 00000100:00100000:1.0:1789760781.631569:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770877739239 time 770916730305 state 0x0 ref 1 00000100:00100000:1.0:1789760781.631589:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770916990721 time 770916730305 state 0x1 ref 1 00000100:00100000:1.0:1789760781.631626:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770916990721 time 770962556314 state 0x0 ref 1 00000100:00100000:1.0:1789760781.631639:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484343680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760781.631706:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24760320, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:1.0:1789760781.631718:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.631738:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:1.0:1789760781.631743:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760781.631751:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24760320->24764415] (req 24760320->24764415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.631767:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24760320->24764415] (req 24760320->24764415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760781.659764:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973363 is committed 00002000:00010000:3.0:1789760781.660324:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24760320 -> 24764416 00010000:00010000:3.0:1789760781.660422:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24760320->24764415] (req 24760320->24764415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.660440:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24760320->24764415] (req 24760320->24764415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.660450:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24760320->24764415] (req 24760320->24764415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.660464:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.660469:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24760320->24764415] (req 24760320->24764415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.660480:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24760320->24764415] (req 24760320->24764415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.660505:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24760320->24764415] (req 24760320->24764415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.660514:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf3826396209f08 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24760320->24764415] (req 24760320->24764415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760781.660646:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484343680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29001us (29276us total) trans 4294973363 rc 0/0 00000100:00100000:3.0:1789760781.660660:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6074 00000100:00100000:3.0:1789760781.660672:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770916990721 time 770962556314 state 0x2 ref 0 00000100:00100000:0.0:1789760781.664522:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760781.664740:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484343808 00000100:00100000:2.0:1789760781.664753:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484343808 00000100:00100000:2.0:1789760781.664838:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770916990721 time 770962556314 state 0x0 ref 1 00000100:00100000:2.0:1789760781.664856:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770962816730 time 770962556314 state 0x1 ref 1 00000100:00100000:2.0:1789760781.664930:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770962816730 time 770995860692 state 0x0 ref 1 00000100:00100000:2.0:1789760781.664943:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484343808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760781.665008:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24764416, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:2.0:1789760781.665030:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.665051:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=770s v=(0 0) 00010000:00010000:2.0:1789760781.665056:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760781.665108:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24764416->24768511] (req 24764416->24768511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.665127:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24764416->24768511] (req 24764416->24768511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760781.690535:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973364 is committed 00002000:00010000:0.0:1789760781.690654:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24764416 -> 24768512 00010000:00010000:0.0:1789760781.690673:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24764416->24768511] (req 24764416->24768511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.690689:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24764416->24768511] (req 24764416->24768511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.690700:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24764416->24768511] (req 24764416->24768511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.690708:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760781.690714:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24764416->24768511] (req 24764416->24768511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.690725:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24764416->24768511] (req 24764416->24768511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.690754:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24764416->24768511] (req 24764416->24768511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.690763:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf3826396209f0f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24764416->24768511] (req 24764416->24768511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760781.690887:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484343808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25938us (26372us total) trans 4294973364 rc 0/0 00000100:00100000:0.0:1789760781.690901:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6075 00000100:00100000:0.0:1789760781.690914:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770962816730 time 770995860692 state 0x2 ref 0 00000100:00100000:0.0:1789760781.693528:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760781.693680:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484343936 00000100:00100000:3.0:1789760781.693693:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484343936 00000100:00100000:3.0:1789760781.693764:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770962816730 time 770995860692 state 0x0 ref 1 00000100:00100000:3.0:1789760781.693784:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770996121108 time 770995860692 state 0x1 ref 1 00000100:00100000:3.0:1789760781.693874:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770996121108 time 771024803682 state 0x0 ref 1 00000100:00100000:3.0:1789760781.693893:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484343936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760781.693956:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24768512, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:3.0:1789760781.693967:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.693988:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:3.0:1789760781.693993:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760781.694002:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24768512->24772607] (req 24768512->24772607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.694028:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24768512->24772607] (req 24768512->24772607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760781.721594:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973365 is committed 00002000:00010000:3.0:1789760781.721776:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24768512 -> 24772608 00010000:00010000:3.0:1789760781.721863:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24768512->24772607] (req 24768512->24772607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.721881:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24768512->24772607] (req 24768512->24772607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.721892:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24768512->24772607] (req 24768512->24772607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.721901:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.721908:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24768512->24772607] (req 24768512->24772607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.721921:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24768512->24772607] (req 24768512->24772607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.721949:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24768512->24772607] (req 24768512->24772607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.721958:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f16 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24768512->24772607] (req 24768512->24772607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760781.722141:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484343936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28243us (28619us total) trans 4294973365 rc 0/0 00000100:00100000:3.0:1789760781.722154:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6076 00000100:00100000:3.0:1789760781.722167:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770996121108 time 771024803682 state 0x2 ref 0 00000100:00100000:2.0:1789760781.736598:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760781.736717:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484344064 00000100:00100000:0.0:1789760781.736730:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484344064 00000100:00100000:0.0:1789760781.736875:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 770996121108 time 771024803682 state 0x0 ref 1 00000100:00100000:0.0:1789760781.736895:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771025064098 time 771024803682 state 0x1 ref 1 00000100:00100000:0.0:1789760781.736971:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771025064098 time 771067900471 state 0x0 ref 1 00000100:00100000:0.0:1789760781.736988:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484344064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760781.737112:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24772608, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:0.0:1789760781.737123:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.737144:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:0.0:1789760781.737149:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760781.737157:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24772608->24776703] (req 24772608->24776703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.737174:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24772608->24776703] (req 24772608->24776703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760781.780147:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24772608 -> 24776704 00010000:00010000:0.0:1789760781.780177:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24772608->24776703] (req 24772608->24776703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.780192:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24772608->24776703] (req 24772608->24776703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.780207:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24772608->24776703] (req 24772608->24776703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.780214:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760781.780220:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24772608->24776703] (req 24772608->24776703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.780230:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24772608->24776703] (req 24772608->24776703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.780256:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24772608->24776703] (req 24772608->24776703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.780264:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f1d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24772608->24776703] (req 24772608->24776703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760781.780393:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484344064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43402us (43804us total) trans 4294973366 rc 0/0 00000100:00100000:0.0:1789760781.780407:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6077 00000100:00100000:0.0:1789760781.780419:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771025064098 time 771067900471 state 0x2 ref 0 00000001:00080000:2.0:1789760781.782104:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973366 is committed 00000100:00100000:3.0:1789760781.785353:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760781.785468:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484344192 00000100:00100000:1.0:1789760781.785481:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484344192 00000100:00100000:1.0:1789760781.785554:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771025064098 time 771067900471 state 0x0 ref 1 00000100:00100000:1.0:1789760781.785574:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771068160887 time 771067900471 state 0x1 ref 1 00000100:00100000:1.0:1789760781.785648:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771068160887 time 771116578038 state 0x0 ref 1 00000100:00100000:1.0:1789760781.785780:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484344192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760781.785849:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24776704, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:1.0:1789760781.785860:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.785878:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:1.0:1789760781.785883:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760781.785891:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24776704->24780799] (req 24776704->24780799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.785908:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24776704->24780799] (req 24776704->24780799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760781.814367:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973367 is committed 00002000:00010000:3.0:1789760781.814573:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24776704 -> 24780800 00010000:00010000:3.0:1789760781.814593:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24776704->24780799] (req 24776704->24780799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.814608:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24776704->24780799] (req 24776704->24780799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.814618:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24776704->24780799] (req 24776704->24780799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.814633:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760781.814638:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24776704->24780799] (req 24776704->24780799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.814649:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24776704->24780799] (req 24776704->24780799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.814673:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24776704->24780799] (req 24776704->24780799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.814682:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf3826396209f24 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24776704->24780799] (req 24776704->24780799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760781.814861:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484344192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29132us (29452us total) trans 4294973367 rc 0/0 00000100:00100000:3.0:1789760781.814874:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6078 00000100:00100000:3.0:1789760781.814888:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771068160887 time 771116578038 state 0x2 ref 0 00000100:00100000:3.0:1789760781.818322:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760781.818651:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484344320 00000100:00100000:2.0:1789760781.818664:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484344320 00000100:00100000:2.0:1789760781.818736:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771068160887 time 771116578038 state 0x0 ref 1 00000100:00100000:2.0:1789760781.818756:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771116838454 time 771116578038 state 0x1 ref 1 00000100:00100000:2.0:1789760781.818794:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771116838454 time 771149724726 state 0x0 ref 1 00000100:00100000:2.0:1789760781.818813:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484344320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760781.818885:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24780800, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:2.0:1789760781.818897:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.818918:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:2.0:1789760781.818924:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760781.818931:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24780800->24784895] (req 24780800->24784895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760781.818948:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24780800->24784895] (req 24780800->24784895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760781.858748:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24780800 -> 24784896 00010000:00010000:0.0:1789760781.858778:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24780800->24784895] (req 24780800->24784895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.858863:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24780800->24784895] (req 24780800->24784895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.858875:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24780800->24784895] (req 24780800->24784895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.858884:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760781.858890:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24780800->24784895] (req 24780800->24784895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.858900:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24780800->24784895] (req 24780800->24784895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.858926:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24780800->24784895] (req 24780800->24784895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.858935:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf3826396209f2b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24780800->24784895] (req 24780800->24784895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760781.859109:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484344320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40247us (40758us total) trans 4294973368 rc 0/0 00000100:00100000:0.0:1789760781.859124:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6079 00000100:00100000:0.0:1789760781.859137:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771116838454 time 771149724726 state 0x2 ref 0 00000001:00080000:1.0:1789760781.859645:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973368 is committed 00000100:00100000:2.0:1789760781.867832:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760781.867983:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484344448 00000100:00100000:3.0:1789760781.867997:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484344448 00000100:00100000:3.0:1789760781.868414:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771116838454 time 771149724726 state 0x0 ref 1 00000100:00100000:3.0:1789760781.868432:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771149985142 time 771149724726 state 0x1 ref 1 00000100:00100000:3.0:1789760781.868471:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771149985142 time 771199402088 state 0x0 ref 1 00000100:00100000:3.0:1789760781.868484:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484344448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760781.868542:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24784896, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:3.0:1789760781.868551:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.868570:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:3.0:1789760781.868575:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760781.868582:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24784896->24788991] (req 24784896->24788991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760781.868597:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24784896->24788991] (req 24784896->24788991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760781.907199:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973369 is committed 00002000:00010000:1.0:1789760781.907392:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24784896 -> 24788992 00010000:00010000:1.0:1789760781.907413:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24784896->24788991] (req 24784896->24788991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.907433:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24784896->24788991] (req 24784896->24788991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.907443:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24784896->24788991] (req 24784896->24788991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.907450:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760781.907456:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24784896->24788991] (req 24784896->24788991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.907466:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24784896->24788991] (req 24784896->24788991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.907491:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24784896->24788991] (req 24784896->24788991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.907500:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf3826396209f32 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24784896->24788991] (req 24784896->24788991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760781.907620:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484344448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39130us (39795us total) trans 4294973369 rc 0/0 00000100:00100000:1.0:1789760781.907632:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6080 00000100:00100000:1.0:1789760781.907644:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771149985142 time 771199402088 state 0x2 ref 0 00000100:00100000:3.0:1789760781.913565:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760781.916140:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484344576 00000100:00100000:0.0:1789760781.916161:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484344576 00000100:00100000:0.0:1789760781.916269:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771149985142 time 771199402088 state 0x0 ref 1 00000100:00100000:0.0:1789760781.916290:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771199662504 time 771199402088 state 0x1 ref 1 00000100:00100000:0.0:1789760781.916334:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771199662504 time 771247264287 state 0x0 ref 1 00000100:00100000:0.0:1789760781.916350:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484344576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760781.916412:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24788992, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:0.0:1789760781.916423:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.916443:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:0.0:1789760781.916448:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760781.916456:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24788992->24793087] (req 24788992->24793087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.916472:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24788992->24793087] (req 24788992->24793087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760781.962593:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973370 is committed 00002000:00010000:0.0:1789760781.962874:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24788992 -> 24793088 00010000:00010000:0.0:1789760781.962894:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24788992->24793087] (req 24788992->24793087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.962908:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24788992->24793087] (req 24788992->24793087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.962924:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24788992->24793087] (req 24788992->24793087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.962934:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760781.962940:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24788992->24793087] (req 24788992->24793087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.962952:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24788992->24793087] (req 24788992->24793087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.962985:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24788992->24793087] (req 24788992->24793087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760781.962995:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f39 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24788992->24793087] (req 24788992->24793087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760781.963167:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484344576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46814us (49610us total) trans 4294973370 rc 0/0 00000100:00100000:0.0:1789760781.963180:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6081 00000100:00100000:0.0:1789760781.963193:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771199662504 time 771247264287 state 0x2 ref 0 00000100:00100000:3.0:1789760781.967766:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760781.969241:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484344704 00000100:00100000:1.0:1789760781.969261:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484344704 00000100:00100000:1.0:1789760781.969348:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771199662504 time 771247264287 state 0x0 ref 1 00000100:00100000:1.0:1789760781.969368:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771247524703 time 771247264287 state 0x1 ref 1 00000100:00100000:1.0:1789760781.969478:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771247524703 time 771300406949 state 0x0 ref 1 00000100:00100000:1.0:1789760781.969501:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484344704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760781.969572:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24793088, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:1.0:1789760781.969583:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.969604:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:1.0:1789760781.969610:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760781.969617:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24793088->24797183] (req 24793088->24797183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760781.969635:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24793088->24797183] (req 24793088->24797183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760782.007561:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973371 is committed 00002000:00010000:3.0:1789760782.007579:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24793088 -> 24797184 00010000:00010000:3.0:1789760782.007601:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24793088->24797183] (req 24793088->24797183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.007616:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24793088->24797183] (req 24793088->24797183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.007628:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24793088->24797183] (req 24793088->24797183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.007637:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760782.007644:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24793088->24797183] (req 24793088->24797183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.007655:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24793088->24797183] (req 24793088->24797183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.007680:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24793088->24797183] (req 24793088->24797183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.007688:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf3826396209f40 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24793088->24797183] (req 24793088->24797183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760782.007814:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484344704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38310us (40056us total) trans 4294973371 rc 0/0 00000100:00100000:3.0:1789760782.007826:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6082 00000100:00100000:3.0:1789760782.007839:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771247524703 time 771300406949 state 0x2 ref 0 00000100:00100000:3.0:1789760782.012366:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760782.012636:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484344832 00000100:00100000:0.0:1789760782.012649:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484344832 00000100:00100000:0.0:1789760782.012731:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771247524703 time 771300406949 state 0x0 ref 1 00000100:00100000:0.0:1789760782.012745:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771300667365 time 771300406949 state 0x1 ref 1 00000100:00100000:0.0:1789760782.012772:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771300667365 time 771343703680 state 0x0 ref 1 00000100:00100000:0.0:1789760782.012790:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484344832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760782.012833:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24797184, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:0.0:1789760782.012841:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.012856:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:0.0:1789760782.012860:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760782.012865:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24797184->24801279] (req 24797184->24801279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.012878:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24797184->24801279] (req 24797184->24801279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760782.013932:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:3.0:1789760782.060887:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973372 is committed 00002000:00010000:0.0:1789760782.061240:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24797184 -> 24801280 00010000:00010000:0.0:1789760782.061264:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24797184->24801279] (req 24797184->24801279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.061280:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24797184->24801279] (req 24797184->24801279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.061292:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24797184->24801279] (req 24797184->24801279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.061303:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760782.061310:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24797184->24801279] (req 24797184->24801279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.061321:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24797184->24801279] (req 24797184->24801279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.061351:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24797184->24801279] (req 24797184->24801279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.061359:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f47 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24797184->24801279] (req 24797184->24801279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760782.061566:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484344832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48771us (49203us total) trans 4294973372 rc 0/0 00000100:00100000:0.0:1789760782.061580:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6083 00000100:00100000:0.0:1789760782.061594:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771300667365 time 771343703680 state 0x2 ref 0 00000100:00100000:1.0:1789760782.064144:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760782.064281:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484344960 00000100:00100000:3.0:1789760782.064295:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484344960 00000100:00100000:3.0:1789760782.064369:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771300667365 time 771343703680 state 0x0 ref 1 00000100:00100000:3.0:1789760782.064392:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771343964096 time 771343703680 state 0x1 ref 1 00000100:00100000:3.0:1789760782.064427:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771343964096 time 771395357759 state 0x0 ref 1 00000100:00100000:3.0:1789760782.064441:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484344960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760782.064523:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24801280, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:3.0:1789760782.064535:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.064558:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:3.0:1789760782.064564:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760782.064573:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24801280->24805375] (req 24801280->24805375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.064594:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24801280->24805375] (req 24801280->24805375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760782.114405:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24801280 -> 24805376 00010000:00010000:1.0:1789760782.114431:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24801280->24805375] (req 24801280->24805375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.114447:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24801280->24805375] (req 24801280->24805375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.114462:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24801280->24805375] (req 24801280->24805375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.114469:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760782.114475:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24801280->24805375] (req 24801280->24805375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.114484:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24801280->24805375] (req 24801280->24805375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.114507:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24801280->24805375] (req 24801280->24805375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.114515:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf3826396209f4e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24801280->24805375] (req 24801280->24805375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760782.114699:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484344960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50253us (50561us total) trans 4294973373 rc 0/0 00000100:00100000:1.0:1789760782.114712:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6084 00000100:00100000:1.0:1789760782.114723:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771343964096 time 771395357759 state 0x2 ref 0 00000001:00080000:2.0:1789760782.114942:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973373 is committed 00000100:00100000:3.0:1789760782.121255:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760782.121369:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484345088 00000100:00100000:1.0:1789760782.121383:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484345088 00000100:00100000:1.0:1789760782.121452:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771343964096 time 771395357759 state 0x0 ref 1 00000100:00100000:1.0:1789760782.121470:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771395618175 time 771395357759 state 0x1 ref 1 00000100:00100000:1.0:1789760782.121506:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771395618175 time 771452436585 state 0x0 ref 1 00000100:00100000:1.0:1789760782.121519:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484345088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760782.121584:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24805376, mtime: 0x6aad950d, atime: 0x0, ctime: 0x6aad950d, blocks: 0xc007 00010000:00010000:1.0:1789760782.121594:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.121614:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:1.0:1789760782.121619:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760782.121627:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24805376->24809471] (req 24805376->24809471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.121643:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24805376->24809471] (req 24805376->24809471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760782.151874:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24805376 -> 24809472 00002000:00010000:1.0:1789760782.151886:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760781 -> 1789760782 00002000:00010000:1.0:1789760782.151889:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760781 -> 1789760782 00010000:00010000:1.0:1789760782.151906:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24805376->24809471] (req 24805376->24809471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.151922:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24805376->24809471] (req 24805376->24809471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.151933:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24805376->24809471] (req 24805376->24809471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.151942:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760782.151947:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24805376->24809471] (req 24805376->24809471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.151958:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24805376->24809471] (req 24805376->24809471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.151983:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24805376->24809471] (req 24805376->24809471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.151991:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f55 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24805376->24809471] (req 24805376->24809471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760782.152159:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484345088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30635us (30912us total) trans 4294973374 rc 0/0 00000100:00100000:1.0:1789760782.152172:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6085 00000100:00100000:1.0:1789760782.152185:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771395618175 time 771452436585 state 0x2 ref 0 00000001:00080000:2.0:1789760782.152619:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973374 is committed 00000100:00100000:3.0:1789760782.155595:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760782.159857:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484345216 00000100:00100000:1.0:1789760782.159877:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484345216 00000100:00100000:1.0:1789760782.159955:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771395618175 time 771452436585 state 0x0 ref 1 00000100:00100000:1.0:1789760782.160132:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771452697001 time 771452436585 state 0x1 ref 1 00000100:00100000:1.0:1789760782.160209:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771452697001 time 771491139136 state 0x0 ref 1 00000100:00100000:1.0:1789760782.160229:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484345216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760782.160290:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24809472, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:1.0:1789760782.160301:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.160321:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:1.0:1789760782.160327:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760782.160334:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24809472->24813567] (req 24809472->24813567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.160351:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24809472->24813567] (req 24809472->24813567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760782.195044:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973375 is committed 00002000:00010000:2.0:1789760782.199266:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24809472 -> 24813568 00010000:00010000:2.0:1789760782.199361:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24809472->24813567] (req 24809472->24813567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.199377:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24809472->24813567] (req 24809472->24813567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.199389:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24809472->24813567] (req 24809472->24813567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.199399:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760782.199405:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24809472->24813567] (req 24809472->24813567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.199417:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24809472->24813567] (req 24809472->24813567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.199442:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24809472->24813567] (req 24809472->24813567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.199452:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f5c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24809472->24813567] (req 24809472->24813567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760782.199838:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f1f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484345216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39603us (44246us total) trans 4294973375 rc 0/0 00000100:00100000:2.0:1789760782.199899:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6086 00000100:00100000:2.0:1789760782.199913:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771452697001 time 771491139136 state 0x2 ref 0 00000100:00100000:1.0:1789760782.203870:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760782.204097:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484345344 00000100:00100000:0.0:1789760782.204110:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484345344 00000100:00100000:0.0:1789760782.204188:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771452697001 time 771491139136 state 0x0 ref 1 00000100:00100000:0.0:1789760782.204207:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771491399552 time 771491139136 state 0x1 ref 1 00000100:00100000:0.0:1789760782.204290:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771491399552 time 771535219757 state 0x0 ref 1 00000100:00100000:0.0:1789760782.204306:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484345344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760782.204374:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24813568, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:0.0:1789760782.204386:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.204415:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:0.0:1789760782.204421:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760782.204428:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24813568->24817663] (req 24813568->24817663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.204446:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24813568->24817663] (req 24813568->24817663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.221553:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760782.259177:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973376 is committed 00002000:00010000:1.0:1789760782.259454:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24813568 -> 24817664 00010000:00010000:1.0:1789760782.259475:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24813568->24817663] (req 24813568->24817663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.259567:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24813568->24817663] (req 24813568->24817663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.259578:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24813568->24817663] (req 24813568->24817663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.259586:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760782.259593:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24813568->24817663] (req 24813568->24817663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.259604:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24813568->24817663] (req 24813568->24817663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.259631:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24813568->24817663] (req 24813568->24817663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.259639:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf3826396209f63 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24813568->24817663] (req 24813568->24817663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760782.259886:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484345344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55575us (56033us total) trans 4294973376 rc 0/0 00000100:00100000:1.0:1789760782.259899:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6087 00000100:00100000:1.0:1789760782.259911:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771491399552 time 771535219757 state 0x2 ref 0 00000100:00100000:3.0:1789760782.270590:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760782.270891:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484345472 00000100:00100000:1.0:1789760782.270905:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484345472 00000100:00100000:1.0:1789760782.270978:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771491399552 time 771535219757 state 0x0 ref 1 00000100:00100000:1.0:1789760782.270996:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771535480173 time 771535219757 state 0x1 ref 1 00000100:00100000:1.0:1789760782.271047:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771535480173 time 771601977103 state 0x0 ref 1 00000100:00100000:1.0:1789760782.271061:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484345472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760782.271281:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24817664, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:1.0:1789760782.271293:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.271313:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:1.0:1789760782.271319:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760782.271328:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24817664->24821759] (req 24817664->24821759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.271345:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24817664->24821759] (req 24817664->24821759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760782.312628:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24817664 -> 24821760 00010000:00010000:1.0:1789760782.312656:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24817664->24821759] (req 24817664->24821759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.312671:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24817664->24821759] (req 24817664->24821759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.312682:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24817664->24821759] (req 24817664->24821759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.312690:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760782.312696:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24817664->24821759] (req 24817664->24821759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.312706:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24817664->24821759] (req 24817664->24821759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.312730:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24817664->24821759] (req 24817664->24821759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.312739:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf3826396209f6a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24817664->24821759] (req 24817664->24821759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760782.312873:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484345472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41808us (42291us total) trans 4294973377 rc 0/0 00000100:00100000:1.0:1789760782.312886:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6088 00000100:00100000:1.0:1789760782.312898:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771535480173 time 771601977103 state 0x2 ref 0 00000001:00080000:0.0:1789760782.315709:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973377 is committed 00000100:00100000:2.0:1789760782.316579:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760782.316683:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484345600 00000100:00100000:1.0:1789760782.316696:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484345600 00000100:00100000:1.0:1789760782.316772:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771535480173 time 771601977103 state 0x0 ref 1 00000100:00100000:1.0:1789760782.316857:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771602237519 time 771601977103 state 0x1 ref 1 00000100:00100000:1.0:1789760782.316939:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771602237519 time 771647868216 state 0x0 ref 1 00000100:00100000:1.0:1789760782.316956:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484345600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760782.317030:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24821760, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:1.0:1789760782.317040:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.317059:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:1.0:1789760782.317087:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760782.317105:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24821760->24825855] (req 24821760->24825855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.317131:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24821760->24825855] (req 24821760->24825855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760782.350629:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973378 is committed 00002000:00010000:2.0:1789760782.350773:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24821760 -> 24825856 00010000:00010000:2.0:1789760782.350804:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24821760->24825855] (req 24821760->24825855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.350821:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24821760->24825855] (req 24821760->24825855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.350832:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24821760->24825855] (req 24821760->24825855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.350841:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760782.350846:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24821760->24825855] (req 24821760->24825855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.350858:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24821760->24825855] (req 24821760->24825855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.350883:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24821760->24825855] (req 24821760->24825855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.350891:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf3826396209f71 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24821760->24825855] (req 24821760->24825855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760782.351423:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484345600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34048us (34431us total) trans 4294973378 rc 0/0 00000100:00100000:2.0:1789760782.351436:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6089 00000100:00100000:2.0:1789760782.351449:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771602237519 time 771647868216 state 0x2 ref 0 00000100:00100000:3.0:1789760782.355416:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760782.355919:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484345728 00000100:00100000:2.0:1789760782.355933:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484345728 00000100:00100000:2.0:1789760782.356008:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771602237519 time 771647868216 state 0x0 ref 1 00000100:00100000:2.0:1789760782.356036:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771648128632 time 771647868216 state 0x1 ref 1 00000100:00100000:2.0:1789760782.356310:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771648128632 time 771687238793 state 0x0 ref 1 00000100:00100000:2.0:1789760782.356327:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484345728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760782.356394:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24825856, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:2.0:1789760782.356405:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.356426:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:2.0:1789760782.356432:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760782.356440:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24825856->24829951] (req 24825856->24829951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.356458:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24825856->24829951] (req 24825856->24829951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760782.393522:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973379 is committed 00002000:00010000:2.0:1789760782.393643:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24825856 -> 24829952 00010000:00010000:2.0:1789760782.393664:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24825856->24829951] (req 24825856->24829951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.393678:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24825856->24829951] (req 24825856->24829951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.393690:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24825856->24829951] (req 24825856->24829951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.393699:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760782.393705:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24825856->24829951] (req 24825856->24829951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.393715:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24825856->24829951] (req 24825856->24829951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.393741:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24825856->24829951] (req 24825856->24829951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.393749:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf3826396209f78 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24825856->24829951] (req 24825856->24829951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760782.393879:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484345728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37549us (38471us total) trans 4294973379 rc 0/0 00000100:00100000:2.0:1789760782.393891:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6090 00000100:00100000:2.0:1789760782.393903:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771648128632 time 771687238793 state 0x2 ref 0 00000100:00100000:2.0:1789760782.403244:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760782.403313:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484345856 00000100:00100000:0.0:1789760782.403326:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484345856 00000100:00100000:0.0:1789760782.403414:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771648128632 time 771687238793 state 0x0 ref 1 00000100:00100000:0.0:1789760782.403435:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771687499209 time 771687238793 state 0x1 ref 1 00000100:00100000:0.0:1789760782.403472:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771687499209 time 771734401706 state 0x0 ref 1 00000100:00100000:0.0:1789760782.403488:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484345856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760782.403561:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24829952, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:0.0:1789760782.403573:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.403597:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:0.0:1789760782.403602:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760782.403612:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24829952->24834047] (req 24829952->24834047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.403630:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24829952->24834047] (req 24829952->24834047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760782.432458:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973380 is committed 00002000:00010000:3.0:1789760782.432620:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24829952 -> 24834048 00010000:00010000:3.0:1789760782.432641:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24829952->24834047] (req 24829952->24834047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.432656:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24829952->24834047] (req 24829952->24834047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.432666:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24829952->24834047] (req 24829952->24834047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.432685:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760782.432691:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24829952->24834047] (req 24829952->24834047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.432701:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24829952->24834047] (req 24829952->24834047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.432726:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24829952->24834047] (req 24829952->24834047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.432735:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf3826396209f7f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24829952->24834047] (req 24829952->24834047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760782.432860:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484345856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29369us (29624us total) trans 4294973380 rc 0/0 00000100:00100000:3.0:1789760782.432872:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6091 00000100:00100000:3.0:1789760782.432884:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771687499209 time 771734401706 state 0x2 ref 0 00000100:00100000:2.0:1789760782.435665:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760782.435871:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484345984 00000100:00100000:1.0:1789760782.435893:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484345984 00000100:00100000:1.0:1789760782.435966:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771687499209 time 771734401706 state 0x0 ref 1 00000100:00100000:1.0:1789760782.435986:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771734662122 time 771734401706 state 0x1 ref 1 00000100:00100000:2.0:1789760782.436604:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771734662122 time 771767530564 state 0x0 ref 1 00000100:00100000:2.0:1789760782.436626:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484345984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760782.436692:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24834048, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:2.0:1789760782.436702:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.436722:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:2.0:1789760782.436727:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760782.436734:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24834048->24838143] (req 24834048->24838143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.436751:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24834048->24838143] (req 24834048->24838143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760782.474292:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973381 is committed 00002000:00010000:2.0:1789760782.474437:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24834048 -> 24838144 00010000:00010000:2.0:1789760782.474456:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24834048->24838143] (req 24834048->24838143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.474471:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24834048->24838143] (req 24834048->24838143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.474480:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24834048->24838143] (req 24834048->24838143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.474489:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760782.474495:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24834048->24838143] (req 24834048->24838143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.474506:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24834048->24838143] (req 24834048->24838143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.474531:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24834048->24838143] (req 24834048->24838143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.474538:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf3826396209f86 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24834048->24838143] (req 24834048->24838143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760782.474666:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484345984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38039us (39011us total) trans 4294973381 rc 0/0 00000100:00100000:2.0:1789760782.474678:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6092 00000100:00100000:2.0:1789760782.474691:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771734662122 time 771767530564 state 0x2 ref 0 00000100:00100000:2.0:1789760782.481340:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760782.481881:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484346112 00000100:00100000:1.0:1789760782.481895:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484346112 00000100:00100000:1.0:1789760782.481973:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771734662122 time 771767530564 state 0x0 ref 1 00000100:00100000:1.0:1789760782.481993:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771767790980 time 771767530564 state 0x1 ref 1 00000100:00100000:1.0:1789760782.482035:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771767790980 time 771812965639 state 0x0 ref 1 00000100:00100000:1.0:1789760782.482052:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484346112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760782.482181:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24838144, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:1.0:1789760782.482192:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.482212:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:1.0:1789760782.482218:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760782.482225:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24838144->24842239] (req 24838144->24842239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.482243:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24838144->24842239] (req 24838144->24842239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760782.517682:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973382 is committed 00002000:00010000:1.0:1789760782.517812:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24838144 -> 24842240 00010000:00010000:1.0:1789760782.517834:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24838144->24842239] (req 24838144->24842239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.517850:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24838144->24842239] (req 24838144->24842239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.517861:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24838144->24842239] (req 24838144->24842239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.517869:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760782.517875:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24838144->24842239] (req 24838144->24842239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.517886:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24838144->24842239] (req 24838144->24842239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.517917:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24838144->24842239] (req 24838144->24842239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.517926:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf3826396209f8d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24838144->24842239] (req 24838144->24842239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760782.518127:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484346112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36072us (36795us total) trans 4294973382 rc 0/0 00000100:00100000:1.0:1789760782.518141:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6093 00000100:00100000:1.0:1789760782.518154:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771767790980 time 771812965639 state 0x2 ref 0 00000100:00100000:1.0:1789760782.523482:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760782.523569:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484346240 00000100:00100000:3.0:1789760782.523583:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484346240 00000100:00100000:3.0:1789760782.523663:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771767790980 time 771812965639 state 0x0 ref 1 00000100:00100000:3.0:1789760782.523683:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771813226055 time 771812965639 state 0x1 ref 1 00000100:00100000:3.0:1789760782.523721:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771813226055 time 771854651610 state 0x0 ref 1 00000100:00100000:3.0:1789760782.523741:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484346240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760782.523807:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24842240, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:3.0:1789760782.523818:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.523838:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:3.0:1789760782.523843:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760782.523856:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24842240->24846335] (req 24842240->24846335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.523873:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24842240->24846335] (req 24842240->24846335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760782.562817:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973383 is committed 00002000:00010000:0.0:1789760782.563294:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24842240 -> 24846336 00010000:00010000:0.0:1789760782.563314:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24842240->24846335] (req 24842240->24846335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.563340:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24842240->24846335] (req 24842240->24846335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.563350:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24842240->24846335] (req 24842240->24846335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.563360:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760782.563367:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24842240->24846335] (req 24842240->24846335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.563378:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24842240->24846335] (req 24842240->24846335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.563406:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24842240->24846335] (req 24842240->24846335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.563414:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209f94 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24842240->24846335] (req 24842240->24846335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760782.563544:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484346240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39799us (40070us total) trans 4294973383 rc 0/0 00000100:00100000:0.0:1789760782.563555:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6094 00000100:00100000:0.0:1789760782.563568:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771813226055 time 771854651610 state 0x2 ref 0 00000100:00100000:2.0:1789760782.568632:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760782.568703:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484346368 00000100:00100000:3.0:1789760782.568716:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484346368 00000100:00100000:3.0:1789760782.568798:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771813226055 time 771854651610 state 0x0 ref 1 00000100:00100000:3.0:1789760782.568817:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771854912026 time 771854651610 state 0x1 ref 1 00000100:00100000:3.0:1789760782.568852:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771854912026 time 771899781606 state 0x0 ref 1 00000100:00100000:3.0:1789760782.568867:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484346368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760782.568931:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24846336, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:3.0:1789760782.568941:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.569132:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:3.0:1789760782.569139:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760782.569147:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24846336->24850431] (req 24846336->24850431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.569166:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24846336->24850431] (req 24846336->24850431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760782.610872:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973384 is committed 00002000:00010000:3.0:1789760782.610881:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24846336 -> 24850432 00010000:00010000:3.0:1789760782.610903:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24846336->24850431] (req 24846336->24850431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.610919:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24846336->24850431] (req 24846336->24850431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.610929:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24846336->24850431] (req 24846336->24850431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.610938:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760782.610944:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24846336->24850431] (req 24846336->24850431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.610955:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24846336->24850431] (req 24846336->24850431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.610982:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24846336->24850431] (req 24846336->24850431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.610992:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209f9b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24846336->24850431] (req 24846336->24850431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760782.611180:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484346368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42308us (42554us total) trans 4294973384 rc 0/0 00000100:00100000:3.0:1789760782.611195:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6095 00000100:00100000:3.0:1789760782.611209:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771854912026 time 771899781606 state 0x2 ref 0 00000100:00100000:3.0:1789760782.615309:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760782.615388:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484346496 00000100:00100000:2.0:1789760782.615402:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484346496 00000100:00100000:2.0:1789760782.615479:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771854912026 time 771899781606 state 0x0 ref 1 00000100:00100000:2.0:1789760782.615498:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771900042022 time 771899781606 state 0x1 ref 1 00000100:00100000:2.0:1789760782.615599:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771900042022 time 771946528478 state 0x0 ref 1 00000100:00100000:2.0:1789760782.615618:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484346496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760782.615682:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24850432, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:2.0:1789760782.615693:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.615713:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:2.0:1789760782.615717:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760782.615725:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24850432->24854527] (req 24850432->24854527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.615740:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24850432->24854527] (req 24850432->24854527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760782.644902:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973385 is committed 00002000:00010000:2.0:1789760782.645216:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24850432 -> 24854528 00010000:00010000:2.0:1789760782.645231:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24850432->24854527] (req 24850432->24854527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.645244:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24850432->24854527] (req 24850432->24854527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.645252:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24850432->24854527] (req 24850432->24854527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.645260:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760782.645265:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24850432->24854527] (req 24850432->24854527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.645275:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24850432->24854527] (req 24850432->24854527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.645296:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24850432->24854527] (req 24850432->24854527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.645305:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24850432->24854527] (req 24850432->24854527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760782.646574:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484346496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30948us (31272us total) trans 4294973385 rc 0/0 00000100:00100000:2.0:1789760782.646588:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6096 00000100:00100000:2.0:1789760782.646599:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771900042022 time 771946528478 state 0x2 ref 0 00000100:00100000:1.0:1789760782.652553:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760782.653060:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484346624 00000100:00100000:2.0:1789760782.653100:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484346624 00000100:00100000:2.0:1789760782.653161:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771900042022 time 771946528478 state 0x0 ref 1 00000100:00100000:2.0:1789760782.653179:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771946788894 time 771946528478 state 0x1 ref 1 00000100:00100000:2.0:1789760782.653209:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771946788894 time 771984140639 state 0x0 ref 1 00000100:00100000:2.0:1789760782.653222:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484346624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760782.653277:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24854528, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:2.0:1789760782.653285:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.653301:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=771s v=(0 0) 00010000:00010000:2.0:1789760782.653305:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760782.653311:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24854528->24858623] (req 24854528->24858623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.653324:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24854528->24858623] (req 24854528->24858623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760782.687520:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973386 is committed 00002000:00010000:2.0:1789760782.687627:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24854528 -> 24858624 00010000:00010000:2.0:1789760782.687642:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24854528->24858623] (req 24854528->24858623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.687655:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24854528->24858623] (req 24854528->24858623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.687664:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24854528->24858623] (req 24854528->24858623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.687670:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760782.687675:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24854528->24858623] (req 24854528->24858623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.687683:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24854528->24858623] (req 24854528->24858623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.687702:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24854528->24858623] (req 24854528->24858623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.687710:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fa9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24854528->24858623] (req 24854528->24858623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760782.688424:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484346624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35198us (35875us total) trans 4294973386 rc 0/0 00000100:00100000:2.0:1789760782.688436:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6097 00000100:00100000:2.0:1789760782.688446:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771946788894 time 771984140639 state 0x2 ref 0 00000100:00100000:0.0:1789760782.696785:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760782.697038:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484346752 00000100:00100000:3.0:1789760782.697048:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484346752 00000100:00100000:3.0:1789760782.697220:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771946788894 time 771984140639 state 0x0 ref 1 00000100:00100000:3.0:1789760782.697235:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771984401055 time 771984140639 state 0x1 ref 1 00000100:00100000:3.0:1789760782.697267:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771984401055 time 772028198367 state 0x0 ref 1 00000100:00100000:3.0:1789760782.697279:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484346752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760782.697401:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24858624, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:3.0:1789760782.697411:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.697427:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:3.0:1789760782.697432:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760782.697439:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24858624->24862719] (req 24858624->24862719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.697451:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24858624->24862719] (req 24858624->24862719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760782.734424:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e85500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508939008:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.734515:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.734569:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508939520:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.734610:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.734635:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e85880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508940032:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.734979:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.735026:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8910380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508940544:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.735052:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760782.735193:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508940032 00000100:00100000:1.0:1789760782.735201:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508940032 00000100:00100000:0.0:1789760782.735297:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:1.0:1789760782.735298:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508940544 00000100:00100000:1.0:1789760782.735304:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508940544 00000100:00100000:0.0:1789760782.735309:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508939136:0@lo:400:kworker.0 00000100:00100000:1.0:1789760782.735332:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 347 00000100:00100000:0.0:1789760782.735344:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760782.735346:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508940032:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.735367:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e85c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508939648:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.735419:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.735436:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8911880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508940160:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.735457:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1789760782.735489:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:2.0:1789760782.735494:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760782.735545:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8912300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508940672:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760782.735575:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.735625:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e87100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508939264:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.735822:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.735855:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e85180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508939776:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.735887:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760782.735892:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ad500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508940032:12345-0@lo:400:kworker.0 Request processed in 549us (912us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760782.735902:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 347 00000100:00100000:0.0:1789760782.735908:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8910000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508940288:0@lo:400:kworker.0 00000100:00100000:1.0:1789760782.735925:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 348 00000100:00100000:1.0:1789760782.735937:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508940544:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.735939:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.735974:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508940800:0@lo:13:osp-pre-1-0.0 00000100:00100000:1.0:1789760782.735982:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508940544:12345-0@lo:400:kworker.0 Request processed in 49us (928us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760782.735990:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 348 00000100:00100000:0.0:1789760782.736016:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.736039:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [3->2] 00000100:00100000:0.0:1789760782.736046:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0001_UUID:3480:1876699508939392:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.736097:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.736120:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e84700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508939904:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.736145:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760782.736163:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8913b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508940416:0@lo:400:kworker.0 00000001:00080000:2.0:1789760782.736163:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973387 is committed 00000100:00100000:3.0:1789760782.736163:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508939904 00000100:00100000:3.0:1789760782.736170:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508939904 00000100:00100000:0.0:1789760782.736190:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1789760782.736199:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760782.736215:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508940416 00000100:00100000:3.0:1789760782.736218:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508940416 00000100:00100000:0.0:1789760782.736233:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508939008 00000100:00100000:0.0:1789760782.736239:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508939008 00002000:00010000:2.0:1789760782.736244:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24858624 -> 24862720 00000100:00100000:3.0:1789760782.736244:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 349 00000100:00100000:3.0:1789760782.736252:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508939904:12345-0@lo:400:kworker.0 00010000:00010000:2.0:1789760782.736258:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24858624->24862719] (req 24858624->24862719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.736271:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24858624->24862719] (req 24858624->24862719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.736280:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24858624->24862719] (req 24858624->24862719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760782.736286:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 494 00010000:00010000:2.0:1789760782.736294:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00000100:00100000:3.0:1789760782.736295:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3480:x1876699508939904:12345-0@lo:400:kworker.0 Request processed in 44us (150us total) trans 0 rc 0/0 00010000:00010000:2.0:1789760782.736299:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24858624->24862719] (req 24858624->24862719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760782.736300:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8039c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508939008:12345-0@lo:400:kworker.0 00010000:00010000:2.0:1789760782.736321:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24858624->24862719] (req 24858624->24862719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760782.736342:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8039c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508939008:12345-0@lo:400:kworker.0 Request processed in 46us (1831us total) trans 0 rc 0/0 00010000:00010000:2.0:1789760782.736342:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24858624->24862719] (req 24858624->24862719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760782.736357:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 494 00010000:00010000:2.0:1789760782.736361:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf3826396209fb0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24858624->24862719] (req 24858624->24862719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760782.736388:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508939520 00000100:00100000:0.0:1789760782.736393:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508939520 00000100:00100000:0.0:1789760782.736416:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508939264 00000100:00100000:0.0:1789760782.736419:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508939264 00000100:00100000:0.0:1789760782.736433:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 471 00000100:00100000:0.0:1789760782.736439:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508939520:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.736481:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8913100 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508939520:12345-0@lo:400:kworker.0 Request processed in 31us (1860us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760782.736488:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 471 00000100:00100000:0.0:1789760782.736500:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 472 00000100:00100000:0.0:1789760782.736507:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508939264:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760782.736536:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508940672 00000100:00100000:0.0:1789760782.736538:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508939264:12345-0@lo:400:kworker.0 Request processed in 32us (716us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760782.736542:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508940672 00000100:00100000:0.0:1789760782.736548:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 472 00000100:00100000:0.0:1789760782.736573:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 350 00000100:00100000:1.0:1789760782.736576:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508940800 00000100:00100000:1.0:1789760782.736581:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508940800 00000100:00100000:0.0:1789760782.736583:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508940416:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760782.736599:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 412 00000100:00100000:1.0:1789760782.736608:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508940672:12345-0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760782.736630:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f45c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508940416:12345-0@lo:400:kworker.0 Request processed in 50us (440us total) trans 0 rc 0/0 00002000:00000024:1.0:1789760782.736647:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760782.736683:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+6:3478:x1876699508940672:12345-0@lo:13:osp-pre-1-1.0 Request processed in 76us (1107us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760782.736689:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 412 00000100:00100000:1.0:1789760782.736703:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 413 00000100:00100000:1.0:1789760782.736709:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fdb2f050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508940800:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:1.0:1789760782.736718:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760782.736753:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fdb2f050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508940800:12345-0@lo:13:osp-pre-1-0.0 Request processed in 44us (747us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760782.736759:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 413 00000100:00100000:2.0:1789760782.737025:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484346752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39745us (40312us total) trans 4294973387 rc 0/0 00000100:00100000:2.0:1789760782.737035:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6098 00000100:00100000:2.0:1789760782.737201:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771984401055 time 772028198367 state 0x2 ref 0 00000100:00080000:2.0:1789760782.737230:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:0.0:1789760782.737294:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 350 00000100:00100000:0.0:1789760782.737320:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508939136 00000100:00100000:0.0:1789760782.737326:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508939136 00000100:00100000:0.0:1789760782.737350:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508939648 00000100:00100000:0.0:1789760782.737353:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508939648 00000100:00100000:0.0:1789760782.737382:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508940160 00000100:00100000:0.0:1789760782.737386:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508940160 00000100:00100000:0.0:1789760782.737398:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508939776 00000100:00100000:0.0:1789760782.737400:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508939776 00000100:00100000:0.0:1789760782.737411:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508940288 00000100:00100000:0.0:1789760782.737414:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508940288 00000100:00100000:0.0:1789760782.737426:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508939392 00000100:00100000:0.0:1789760782.737430:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508939392 00000100:00100000:0.0:1789760782.737443:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 995 00000100:00100000:0.0:1789760782.737450:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508939136:12345-0@lo:400:kworker.0 00000100:00080000:2.0:1789760782.737468:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760782.737472:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:0.0:1789760782.738270:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508939136:12345-0@lo:400:kworker.0 Request processed in 815us (2923us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760782.738280:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 995 00000100:00100000:0.0:1789760782.738303:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 996 00000100:00100000:0.0:1789760782.738314:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508939648:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.738351:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508939648:12345-0@lo:400:kworker.0 Request processed in 42us (2931us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760782.738357:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 996 00000100:00100000:0.0:1789760782.738369:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 997 00000100:00100000:0.0:1789760782.738376:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508940160:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.738403:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508940160:12345-0@lo:400:kworker.0 Request processed in 29us (2945us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760782.738409:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 997 00000100:00100000:0.0:1789760782.738419:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 998 00000100:00100000:0.0:1789760782.738425:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508939776:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.738449:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defddf80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508939776:12345-0@lo:400:kworker.0 Request processed in 25us (2562us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760782.738455:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 998 00000100:00100000:0.0:1789760782.738466:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 999 00000100:00100000:0.0:1789760782.738535:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508940288:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.738569:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508940288:12345-0@lo:400:kworker.0 Request processed in 97us (2634us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760782.738578:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 999 00000100:00100000:0.0:1789760782.738714:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1000 00000100:00100000:0.0:1789760782.738721:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3480:x1876699508939392:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.738821:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0001_UUID+5:3480:x1876699508939392:12345-0@lo:400:kworker.0 Request processed in 100us (2723us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760782.738827:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1000 00000100:00100000:3.0:1789760782.738909:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 349 00000100:00100000:0.0:1789760782.738999:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0001_UUID:3480:1876699508939392:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739159:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e84700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508939904:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739175:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8913b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508940416:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739187:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e87100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508939264:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739198:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e85180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508939776:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739208:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8910000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508940288:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739229:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508940800:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760782.739287:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508939136:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739300:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e85c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508939648:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739310:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8911880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508940160:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739322:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8912300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508940672:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760782.739331:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e85500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508939008:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739340:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508939520:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739352:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e85880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508940032:0@lo:400:kworker.0 00000100:00100000:0.0:1789760782.739361:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8910380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508940544:0@lo:400:kworker.0 00000100:00100000:3.0:1789760782.744487:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760782.744573:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484346880 00000100:00100000:0.0:1789760782.744583:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484346880 00000100:00100000:0.0:1789760782.744639:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 771984401055 time 772028198367 state 0x0 ref 1 00000100:00100000:0.0:1789760782.744652:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772028458783 time 772028198367 state 0x1 ref 1 00000100:00100000:0.0:1789760782.744738:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772028458783 time 772075667468 state 0x0 ref 1 00000100:00100000:0.0:1789760782.744754:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484346880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760782.744803:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24862720, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:0.0:1789760782.744896:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.744913:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:0.0:1789760782.744918:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760782.744925:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24862720->24866815] (req 24862720->24866815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.744948:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24862720->24866815] (req 24862720->24866815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760782.779262:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973388 is committed 00002000:00010000:1.0:1789760782.779613:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24862720 -> 24866816 00010000:00010000:1.0:1789760782.779630:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24862720->24866815] (req 24862720->24866815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.779644:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24862720->24866815] (req 24862720->24866815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.779653:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24862720->24866815] (req 24862720->24866815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.779661:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760782.779665:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24862720->24866815] (req 24862720->24866815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.779673:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24862720->24866815] (req 24862720->24866815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.779695:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24862720->24866815] (req 24862720->24866815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760782.779703:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf3826396209fb7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24862720->24866815] (req 24862720->24866815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760782.780000:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484346880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35245us (35518us total) trans 4294973388 rc 0/0 00000100:00100000:1.0:1789760782.780022:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6099 00000100:00100000:1.0:1789760782.780032:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772028458783 time 772075667468 state 0x2 ref 0 00000100:00100000:1.0:1789760782.783231:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760782.783959:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484347008 00000100:00100000:2.0:1789760782.783972:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484347008 00000100:00100000:2.0:1789760782.784046:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772028458783 time 772075667468 state 0x0 ref 1 00000100:00100000:2.0:1789760782.784555:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772075927884 time 772075667468 state 0x1 ref 1 00000100:00100000:2.0:1789760782.784594:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772075927884 time 772115524619 state 0x0 ref 1 00000100:00100000:2.0:1789760782.784610:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484347008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760782.784663:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24866816, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:2.0:1789760782.784673:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.784687:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760782.784691:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760782.784697:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24866816->24870911] (req 24866816->24870911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.784710:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24866816->24870911] (req 24866816->24870911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760782.827765:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973389 is committed 00002000:00010000:3.0:1789760782.827923:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24866816 -> 24870912 00010000:00010000:3.0:1789760782.827939:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24866816->24870911] (req 24866816->24870911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.827951:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24866816->24870911] (req 24866816->24870911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.827961:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24866816->24870911] (req 24866816->24870911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.827968:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760782.827974:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24866816->24870911] (req 24866816->24870911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.827982:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24866816->24870911] (req 24866816->24870911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.828003:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24866816->24870911] (req 24866816->24870911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.828021:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf3826396209fbe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24866816->24870911] (req 24866816->24870911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760782.828193:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484347008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43583us (44967us total) trans 4294973389 rc 0/0 00000100:00100000:3.0:1789760782.828203:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6100 00000100:00100000:3.0:1789760782.828213:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772075927884 time 772115524619 state 0x2 ref 0 00000100:00100000:0.0:1789760782.835456:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760782.835525:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484347136 00000100:00100000:2.0:1789760782.835535:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484347136 00000100:00100000:2.0:1789760782.835589:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772075927884 time 772115524619 state 0x0 ref 1 00000100:00100000:2.0:1789760782.835605:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772115785035 time 772115524619 state 0x1 ref 1 00000100:00100000:2.0:1789760782.835638:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772115785035 time 772166569102 state 0x0 ref 1 00000100:00100000:2.0:1789760782.835650:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484347136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760782.835700:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24870912, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:2.0:1789760782.835708:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.835725:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760782.835729:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760782.835736:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24870912->24875007] (req 24870912->24875007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760782.835751:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24870912->24875007] (req 24870912->24875007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760782.902109:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973390 is committed 00002000:00010000:3.0:1789760782.902302:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24870912 -> 24875008 00010000:00010000:3.0:1789760782.902318:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24870912->24875007] (req 24870912->24875007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.902332:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24870912->24875007] (req 24870912->24875007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.902341:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24870912->24875007] (req 24870912->24875007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.902424:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760782.902430:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24870912->24875007] (req 24870912->24875007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.902447:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24870912->24875007] (req 24870912->24875007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.902468:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24870912->24875007] (req 24870912->24875007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.902476:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf3826396209fc5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24870912->24875007] (req 24870912->24875007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760782.902687:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484347136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67034us (67248us total) trans 4294973390 rc 0/0 00000100:00100000:3.0:1789760782.902697:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6101 00000100:00100000:3.0:1789760782.902707:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772115785035 time 772166569102 state 0x2 ref 0 00000100:00100000:3.0:1789760782.920185:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760782.920958:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484347264 00000100:00100000:0.0:1789760782.920969:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484347264 00000100:00100000:0.0:1789760782.921034:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772115785035 time 772166569102 state 0x0 ref 1 00000100:00100000:0.0:1789760782.921050:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772166829518 time 772166569102 state 0x1 ref 1 00000100:00100000:3.0:1789760782.921257:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772166829518 time 772252185006 state 0x0 ref 1 00000100:00100000:3.0:1789760782.921272:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484347264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760782.921325:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24875008, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:3.0:1789760782.921333:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.921349:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:3.0:1789760782.921365:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760782.921372:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24875008->24879103] (req 24875008->24879103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.921384:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24875008->24879103] (req 24875008->24879103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760782.965029:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973391 is committed 00002000:00010000:3.0:1789760782.965198:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24875008 -> 24879104 00010000:00010000:3.0:1789760782.965213:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24875008->24879103] (req 24875008->24879103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.965225:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24875008->24879103] (req 24875008->24879103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.965233:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24875008->24879103] (req 24875008->24879103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.965241:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760782.965246:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24875008->24879103] (req 24875008->24879103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.965256:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24875008->24879103] (req 24875008->24879103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.965275:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24875008->24879103] (req 24875008->24879103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760782.965283:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fcc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24875008->24879103] (req 24875008->24879103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760782.965382:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484347264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44109us (45201us total) trans 4294973391 rc 0/0 00000100:00100000:3.0:1789760782.965393:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6102 00000100:00100000:3.0:1789760782.965404:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772166829518 time 772252185006 state 0x2 ref 0 00000100:00100000:2.0:1789760782.973237:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760782.975265:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484347392 00000100:00100000:1.0:1789760782.975279:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484347392 00000100:00100000:0.0:1789760782.975423:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772166829518 time 772252185006 state 0x0 ref 1 00000100:00100000:0.0:1789760782.975439:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772252445422 time 772252185006 state 0x1 ref 1 00000100:00100000:0.0:1789760782.975498:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772252445422 time 772306428351 state 0x0 ref 1 00000100:00100000:0.0:1789760782.975512:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484347392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760782.975571:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24879104, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:0.0:1789760782.975589:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.975606:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:0.0:1789760782.975611:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760782.975617:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24879104->24883199] (req 24879104->24883199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760782.975631:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24879104->24883199] (req 24879104->24883199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760783.015271:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973392 is committed 00002000:00010000:0.0:1789760783.015348:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24879104 -> 24883200 00010000:00010000:0.0:1789760783.015363:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24879104->24883199] (req 24879104->24883199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.015377:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24879104->24883199] (req 24879104->24883199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.015387:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24879104->24883199] (req 24879104->24883199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.015394:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760783.015399:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24879104->24883199] (req 24879104->24883199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.015408:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24879104->24883199] (req 24879104->24883199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.015427:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24879104->24883199] (req 24879104->24883199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.015435:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf3826396209fd3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24879104->24883199] (req 24879104->24883199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760783.015529:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484347392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40017us (42296us total) trans 4294973392 rc 0/0 00000100:00100000:0.0:1789760783.015539:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6103 00000100:00100000:0.0:1789760783.015550:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772252445422 time 772306428351 state 0x2 ref 0 00000100:00100000:0.0:1789760783.022826:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.023098:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484347520 00000100:00100000:2.0:1789760783.023109:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484347520 00000100:00100000:2.0:1789760783.023160:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772252445422 time 772306428351 state 0x0 ref 1 00000100:00100000:2.0:1789760783.023177:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772306688767 time 772306428351 state 0x1 ref 1 00000100:00100000:2.0:1789760783.023278:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772306688767 time 772354208296 state 0x0 ref 1 00000100:00100000:2.0:1789760783.023292:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484347520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.023337:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24883200, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:2.0:1789760783.023345:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.023361:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760783.023366:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.023372:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24883200->24887295] (req 24883200->24887295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.023384:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24883200->24887295] (req 24883200->24887295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760783.050949:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973393 is committed 00002000:00010000:3.0:1789760783.051169:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24883200 -> 24887296 00010000:00010000:3.0:1789760783.051184:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24883200->24887295] (req 24883200->24887295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.051196:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24883200->24887295] (req 24883200->24887295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.051205:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24883200->24887295] (req 24883200->24887295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.051212:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.051216:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24883200->24887295] (req 24883200->24887295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.051224:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24883200->24887295] (req 24883200->24887295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.051246:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24883200->24887295] (req 24883200->24887295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.051252:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf3826396209fda lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24883200->24887295] (req 24883200->24887295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.051342:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484347520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28049us (28520us total) trans 4294973393 rc 0/0 00000100:00100000:3.0:1789760783.051973:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6104 00000100:00100000:3.0:1789760783.051985:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772306688767 time 772354208296 state 0x2 ref 0 00000100:00100000:3.0:1789760783.054342:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.054612:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484347648 00000100:00100000:2.0:1789760783.054621:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484347648 00000100:00100000:2.0:1789760783.054675:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772306688767 time 772354208296 state 0x0 ref 1 00000100:00100000:2.0:1789760783.054691:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772354468712 time 772354208296 state 0x1 ref 1 00000100:00100000:2.0:1789760783.054723:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772354468712 time 772385654144 state 0x0 ref 1 00000100:00100000:2.0:1789760783.054736:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484347648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.054784:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24887296, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:2.0:1789760783.054805:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.054819:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760783.054823:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.054830:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24887296->24891391] (req 24887296->24891391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.054843:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24887296->24891391] (req 24887296->24891391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760783.076176:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973394 is committed 00002000:00010000:3.0:1789760783.076231:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24887296 -> 24891392 00010000:00010000:3.0:1789760783.076248:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24887296->24891391] (req 24887296->24891391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.076260:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24887296->24891391] (req 24887296->24891391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.076270:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24887296->24891391] (req 24887296->24891391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.076278:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.076282:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24887296->24891391] (req 24887296->24891391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.076298:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24887296->24891391] (req 24887296->24891391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.076317:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24887296->24891391] (req 24887296->24891391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.076325:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf3826396209fe1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24887296->24891391] (req 24887296->24891391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.076410:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484347648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 21672us (22072us total) trans 4294973394 rc 0/0 00000100:00100000:3.0:1789760783.076420:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6105 00000100:00100000:3.0:1789760783.076430:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772354468712 time 772385654144 state 0x2 ref 0 00000100:00100000:2.0:1789760783.080170:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.080241:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484347776 00000100:00100000:3.0:1789760783.080259:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484347776 00000100:00100000:3.0:1789760783.080342:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772354468712 time 772385654144 state 0x0 ref 1 00000100:00100000:3.0:1789760783.080363:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772385914560 time 772385654144 state 0x1 ref 1 00000100:00100000:3.0:1789760783.080400:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772385914560 time 772411330382 state 0x0 ref 1 00000100:00100000:3.0:1789760783.080417:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484347776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760783.080483:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24891392, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:3.0:1789760783.080494:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.080515:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:3.0:1789760783.080521:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760783.080529:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24891392->24895487] (req 24891392->24895487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.080545:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24891392->24895487] (req 24891392->24895487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760783.128235:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973395 is committed 00002000:00010000:3.0:1789760783.128463:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24891392 -> 24895488 00010000:00010000:3.0:1789760783.128479:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24891392->24895487] (req 24891392->24895487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.128492:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24891392->24895487] (req 24891392->24895487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.128501:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24891392->24895487] (req 24891392->24895487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.128508:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.128513:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24891392->24895487] (req 24891392->24895487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.128521:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24891392->24895487] (req 24891392->24895487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.128542:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24891392->24895487] (req 24891392->24895487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.128550:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf3826396209fe8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24891392->24895487] (req 24891392->24895487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.128654:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484347776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48235us (48488us total) trans 4294973395 rc 0/0 00000100:00100000:3.0:1789760783.128665:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6106 00000100:00100000:3.0:1789760783.128675:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772385914560 time 772411330382 state 0x2 ref 0 00000100:00100000:0.0:1789760783.131132:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.131986:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484347904 00000100:00100000:2.0:1789760783.131997:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484347904 00000100:00100000:2.0:1789760783.132062:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772385914560 time 772411330382 state 0x0 ref 1 00000100:00100000:2.0:1789760783.132113:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772411590798 time 772411330382 state 0x1 ref 1 00000100:00100000:2.0:1789760783.132143:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772411590798 time 772463074553 state 0x0 ref 1 00000100:00100000:2.0:1789760783.132157:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484347904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.132278:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24895488, mtime: 0x6aad950e, atime: 0x0, ctime: 0x6aad950e, blocks: 0xc007 00010000:00010000:2.0:1789760783.132288:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.132305:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760783.132308:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.132314:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24895488->24899583] (req 24895488->24899583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.132328:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24895488->24899583] (req 24895488->24899583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760783.162194:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973396 is committed 00002000:00010000:0.0:1789760783.162300:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24895488 -> 24899584 00002000:00010000:0.0:1789760783.162305:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760782 -> 1789760783 00002000:00010000:0.0:1789760783.162307:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760782 -> 1789760783 00010000:00010000:0.0:1789760783.162318:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24895488->24899583] (req 24895488->24899583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.162331:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24895488->24899583] (req 24895488->24899583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.162339:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24895488->24899583] (req 24895488->24899583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.162346:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760783.162351:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24895488->24899583] (req 24895488->24899583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.162360:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24895488->24899583] (req 24895488->24899583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.162379:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24895488->24899583] (req 24895488->24899583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.162386:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf3826396209fef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24895488->24899583] (req 24895488->24899583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760783.162482:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484347904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30325us (31355us total) trans 4294973396 rc 0/0 00000100:00100000:0.0:1789760783.162492:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6107 00000100:00100000:0.0:1789760783.162502:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772411590798 time 772463074553 state 0x2 ref 0 00000100:00100000:0.0:1789760783.165734:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.165827:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484348032 00000100:00100000:2.0:1789760783.165837:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484348032 00000100:00100000:2.0:1789760783.165891:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772411590798 time 772463074553 state 0x0 ref 1 00000100:00100000:2.0:1789760783.165906:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772463334969 time 772463074553 state 0x1 ref 1 00000100:00100000:2.0:1789760783.165930:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772463334969 time 772496861665 state 0x0 ref 1 00000100:00100000:2.0:1789760783.165942:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484348032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.165992:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24899584, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:2.0:1789760783.166001:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.166028:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760783.166032:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.166038:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24899584->24903679] (req 24899584->24903679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.166051:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24899584->24903679] (req 24899584->24903679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760783.190361:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973397 is committed 00002000:00010000:2.0:1789760783.190390:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24899584 -> 24903680 00010000:00010000:2.0:1789760783.190406:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24899584->24903679] (req 24899584->24903679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.190421:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24899584->24903679] (req 24899584->24903679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.190431:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24899584->24903679] (req 24899584->24903679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.190439:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760783.190444:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24899584->24903679] (req 24899584->24903679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.190454:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24899584->24903679] (req 24899584->24903679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.190474:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24899584->24903679] (req 24899584->24903679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.190480:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf3826396209ff6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24899584->24903679] (req 24899584->24903679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760783.190558:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484348032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24609us (24825us total) trans 4294973397 rc 0/0 00000100:00100000:2.0:1789760783.190569:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6108 00000100:00100000:2.0:1789760783.190648:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772463334969 time 772496861665 state 0x2 ref 0 00000100:00100000:3.0:1789760783.193328:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.193372:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484348160 00000100:00100000:2.0:1789760783.193383:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484348160 00000100:00100000:2.0:1789760783.193435:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772463334969 time 772496861665 state 0x0 ref 1 00000100:00100000:2.0:1789760783.193448:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772497122081 time 772496861665 state 0x1 ref 1 00000100:00100000:2.0:1789760783.193471:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772497122081 time 772524402900 state 0x0 ref 1 00000100:00100000:2.0:1789760783.193480:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484348160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.193517:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24903680, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:2.0:1789760783.193523:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.193535:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760783.193538:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.193543:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24903680->24907775] (req 24903680->24907775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.193552:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24903680->24907775] (req 24903680->24907775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760783.220888:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24903680 -> 24907776 00010000:00010000:2.0:1789760783.220909:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24903680->24907775] (req 24903680->24907775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.220922:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24903680->24907775] (req 24903680->24907775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.220930:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24903680->24907775] (req 24903680->24907775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.220937:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760783.220941:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24903680->24907775] (req 24903680->24907775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.220950:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24903680->24907775] (req 24903680->24907775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.220968:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24903680->24907775] (req 24903680->24907775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760783.221237:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973398 is committed 00010000:00010000:2.0:1789760783.221363:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf3826396209ffd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24903680->24907775] (req 24903680->24907775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.221505:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484348160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28021us (28181us total) trans 4294973398 rc 0/0 00000100:00100000:3.0:1789760783.221516:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6109 00000100:00100000:3.0:1789760783.221526:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772497122081 time 772524402900 state 0x2 ref 0 00000100:00100000:0.0:1789760783.227336:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.227766:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484348288 00000100:00100000:3.0:1789760783.227777:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484348288 00000100:00100000:3.0:1789760783.227906:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772497122081 time 772524402900 state 0x0 ref 1 00000100:00100000:3.0:1789760783.228203:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772524663316 time 772524402900 state 0x1 ref 1 00000100:00100000:3.0:1789760783.228236:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772524663316 time 772559166634 state 0x0 ref 1 00000100:00100000:3.0:1789760783.228252:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484348288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760783.228698:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24907776, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:3.0:1789760783.228708:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.228726:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:3.0:1789760783.228730:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760783.228737:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24907776->24911871] (req 24907776->24911871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.228751:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24907776->24911871] (req 24907776->24911871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.244392:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760783.271843:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973399 is committed 00002000:00010000:3.0:1789760783.271966:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24907776 -> 24911872 00010000:00010000:3.0:1789760783.271981:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24907776->24911871] (req 24907776->24911871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.271995:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24907776->24911871] (req 24907776->24911871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.272012:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24907776->24911871] (req 24907776->24911871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.272020:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.272025:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24907776->24911871] (req 24907776->24911871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.272034:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24907776->24911871] (req 24907776->24911871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.272053:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24907776->24911871] (req 24907776->24911871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.272060:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a004 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24907776->24911871] (req 24907776->24911871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.272177:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484348288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43925us (44847us total) trans 4294973399 rc 0/0 00000100:00100000:3.0:1789760783.272187:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6110 00000100:00100000:3.0:1789760783.272198:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772524663316 time 772559166634 state 0x2 ref 0 00000100:00100000:0.0:1789760783.275501:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.275660:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484348416 00000100:00100000:2.0:1789760783.275670:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484348416 00000100:00100000:2.0:1789760783.275721:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772524663316 time 772559166634 state 0x0 ref 1 00000100:00100000:2.0:1789760783.275736:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772559427050 time 772559166634 state 0x1 ref 1 00000100:00100000:2.0:1789760783.275766:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772559427050 time 772606697391 state 0x0 ref 1 00000100:00100000:2.0:1789760783.275779:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484348416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.275836:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24911872, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:2.0:1789760783.275844:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.275860:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760783.275864:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.275870:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24911872->24915967] (req 24911872->24915967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.275883:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24911872->24915967] (req 24911872->24915967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760783.310796:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973400 is committed 00002000:00010000:3.0:1789760783.311051:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24911872 -> 24915968 00010000:00010000:3.0:1789760783.311138:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24911872->24915967] (req 24911872->24915967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.311153:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24911872->24915967] (req 24911872->24915967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.311162:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24911872->24915967] (req 24911872->24915967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.311170:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.311175:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24911872->24915967] (req 24911872->24915967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.311186:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24911872->24915967] (req 24911872->24915967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.311211:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24911872->24915967] (req 24911872->24915967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.311219:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a00b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24911872->24915967] (req 24911872->24915967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760783.311361:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484348416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35580us (35864us total) trans 4294973400 rc 0/0 00000100:00100000:2.0:1789760783.311376:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6111 00000100:00100000:2.0:1789760783.311385:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772559427050 time 772606697391 state 0x2 ref 0 00000100:00100000:2.0:1789760783.315525:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.315647:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484348544 00000100:00100000:3.0:1789760783.315656:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484348544 00000100:00100000:3.0:1789760783.315710:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772559427050 time 772606697391 state 0x0 ref 1 00000100:00100000:3.0:1789760783.315725:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772606957807 time 772606697391 state 0x1 ref 1 00000100:00100000:3.0:1789760783.315758:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772606957807 time 772646689038 state 0x0 ref 1 00000100:00100000:3.0:1789760783.315770:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484348544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760783.315819:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24915968, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:3.0:1789760783.315827:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.315844:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:3.0:1789760783.315849:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760783.315855:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24915968->24920063] (req 24915968->24920063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.315869:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24915968->24920063] (req 24915968->24920063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760783.339584:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973401 is committed 00002000:00010000:2.0:1789760783.339858:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24915968 -> 24920064 00010000:00010000:2.0:1789760783.339873:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24915968->24920063] (req 24915968->24920063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.339886:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24915968->24920063] (req 24915968->24920063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.339895:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24915968->24920063] (req 24915968->24920063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.339900:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760783.339904:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24915968->24920063] (req 24915968->24920063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.339911:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24915968->24920063] (req 24915968->24920063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.339930:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24915968->24920063] (req 24915968->24920063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.339937:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a012 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24915968->24920063] (req 24915968->24920063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760783.340184:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484348544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24411us (24664us total) trans 4294973401 rc 0/0 00000100:00100000:2.0:1789760783.340196:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6112 00000100:00100000:2.0:1789760783.340206:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772606957807 time 772646689038 state 0x2 ref 0 00000100:00100000:0.0:1789760783.346045:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.346838:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484348672 00000100:00100000:3.0:1789760783.346849:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484348672 00000100:00100000:3.0:1789760783.346910:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772606957807 time 772646689038 state 0x0 ref 1 00000100:00100000:3.0:1789760783.346926:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772646949454 time 772646689038 state 0x1 ref 1 00000100:00100000:3.0:1789760783.346960:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772646949454 time 772677890295 state 0x0 ref 1 00000100:00100000:3.0:1789760783.346974:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484348672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760783.347212:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24920064, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:3.0:1789760783.347221:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.347236:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:3.0:1789760783.347240:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760783.347246:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24920064->24924159] (req 24920064->24924159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.347259:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24920064->24924159] (req 24920064->24924159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760783.382448:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24920064 -> 24924160 00010000:00010000:3.0:1789760783.382468:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24920064->24924159] (req 24920064->24924159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.382488:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24920064->24924159] (req 24920064->24924159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.382498:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24920064->24924159] (req 24920064->24924159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.382506:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.382511:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24920064->24924159] (req 24920064->24924159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.382521:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24920064->24924159] (req 24920064->24924159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.382541:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24920064->24924159] (req 24920064->24924159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.382550:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a019 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24920064->24924159] (req 24920064->24924159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.382644:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484348672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35669us (36606us total) trans 4294973402 rc 0/0 00000100:00100000:3.0:1789760783.382655:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6113 00000100:00100000:3.0:1789760783.382664:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772646949454 time 772677890295 state 0x2 ref 0 00000001:00080000:1.0:1789760783.382745:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973402 is committed 00000100:00100000:0.0:1789760783.386938:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.387045:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484348800 00000100:00100000:2.0:1789760783.387055:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484348800 00000100:00100000:2.0:1789760783.387354:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772646949454 time 772677890295 state 0x0 ref 1 00000100:00100000:2.0:1789760783.387371:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772678150711 time 772677890295 state 0x1 ref 1 00000100:00100000:2.0:1789760783.387406:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772678150711 time 772718337072 state 0x0 ref 1 00000100:00100000:2.0:1789760783.387419:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484348800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.387470:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24924160, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:2.0:1789760783.387479:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.387496:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760783.387500:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.387506:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24924160->24928255] (req 24924160->24928255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.387520:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24924160->24928255] (req 24924160->24928255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760783.419196:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973403 is committed 00002000:00010000:0.0:1789760783.419341:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24924160 -> 24928256 00010000:00010000:0.0:1789760783.419357:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24924160->24928255] (req 24924160->24928255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.419370:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24924160->24928255] (req 24924160->24928255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.419381:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24924160->24928255] (req 24924160->24928255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.419388:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760783.419393:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24924160->24928255] (req 24924160->24928255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.419407:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24924160->24928255] (req 24924160->24928255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.419430:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24924160->24928255] (req 24924160->24928255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.419437:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a020 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24924160->24928255] (req 24924160->24928255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760783.419573:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484348800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32151us (32639us total) trans 4294973403 rc 0/0 00000100:00100000:0.0:1789760783.419583:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6114 00000100:00100000:0.0:1789760783.419593:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772678150711 time 772718337072 state 0x2 ref 0 00000100:00100000:0.0:1789760783.427031:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.427252:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484348928 00000100:00100000:2.0:1789760783.427262:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484348928 00000100:00100000:2.0:1789760783.427319:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772678150711 time 772718337072 state 0x0 ref 1 00000100:00100000:2.0:1789760783.427335:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772718597488 time 772718337072 state 0x1 ref 1 00000100:00100000:2.0:1789760783.427366:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772718597488 time 772758296984 state 0x0 ref 1 00000100:00100000:2.0:1789760783.427379:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484348928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.427432:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24928256, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:2.0:1789760783.427442:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.427763:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760783.427768:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.427775:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24928256->24932351] (req 24928256->24932351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.427789:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24928256->24932351] (req 24928256->24932351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760783.468717:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24928256 -> 24932352 00010000:00010000:3.0:1789760783.468742:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24928256->24932351] (req 24928256->24932351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.468759:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24928256->24932351] (req 24928256->24932351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.468769:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24928256->24932351] (req 24928256->24932351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.468778:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.468795:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24928256->24932351] (req 24928256->24932351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.468806:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24928256->24932351] (req 24928256->24932351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.468827:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24928256->24932351] (req 24928256->24932351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.468835:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a027 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24928256->24932351] (req 24928256->24932351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.468951:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484348928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41569us (41941us total) trans 4294973404 rc 0/0 00000100:00100000:3.0:1789760783.468962:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6115 00000100:00100000:3.0:1789760783.468972:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772718597488 time 772758296984 state 0x2 ref 0 00000001:00080000:0.0:1789760783.469138:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973404 is committed 00000100:00100000:3.0:1789760783.478189:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760783.478294:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484349056 00000100:00100000:0.0:1789760783.478304:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484349056 00000100:00100000:0.0:1789760783.478365:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772718597488 time 772758296984 state 0x0 ref 1 00000100:00100000:0.0:1789760783.478381:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772758557400 time 772758296984 state 0x1 ref 1 00000100:00100000:0.0:1789760783.478415:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772758557400 time 772809346333 state 0x0 ref 1 00000100:00100000:0.0:1789760783.478428:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba75180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484349056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760783.478476:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24932352, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:0.0:1789760783.478484:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.478501:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:0.0:1789760783.478505:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760783.478511:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24932352->24936447] (req 24932352->24936447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.478525:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24932352->24936447] (req 24932352->24936447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760783.509724:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24932352 -> 24936448 00010000:00010000:0.0:1789760783.509744:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24932352->24936447] (req 24932352->24936447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.509757:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24932352->24936447] (req 24932352->24936447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.509766:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24932352->24936447] (req 24932352->24936447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.509773:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760783.509778:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24932352->24936447] (req 24932352->24936447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.509796:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24932352->24936447] (req 24932352->24936447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.509816:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24932352->24936447] (req 24932352->24936447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.509823:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a02e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24932352->24936447] (req 24932352->24936447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760783.509921:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba75180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484349056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31490us (31737us total) trans 4294973405 rc 0/0 00000100:00100000:0.0:1789760783.509931:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6116 00000100:00100000:0.0:1789760783.509941:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772758557400 time 772809346333 state 0x2 ref 0 00000001:00080000:1.0:1789760783.510402:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973405 is committed 00000100:00100000:1.0:1789760783.516645:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.516852:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484349184 00000100:00100000:3.0:1789760783.516862:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484349184 00000100:00100000:3.0:1789760783.516922:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772758557400 time 772809346333 state 0x0 ref 1 00000100:00100000:3.0:1789760783.516938:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772809606749 time 772809346333 state 0x1 ref 1 00000100:00100000:3.0:1789760783.516973:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772809606749 time 772847903480 state 0x0 ref 1 00000100:00100000:3.0:1789760783.516985:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484349184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760783.517053:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24936448, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:3.0:1789760783.517184:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.517203:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:3.0:1789760783.517207:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760783.517214:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24936448->24940543] (req 24936448->24940543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.517229:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24936448->24940543] (req 24936448->24940543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760783.575894:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973406 is committed 00002000:00010000:3.0:1789760783.583318:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24936448 -> 24940544 00010000:00010000:3.0:1789760783.583339:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24936448->24940543] (req 24936448->24940543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.583360:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24936448->24940543] (req 24936448->24940543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.583369:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24936448->24940543] (req 24936448->24940543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.583376:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.583381:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24936448->24940543] (req 24936448->24940543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.583390:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24936448->24940543] (req 24936448->24940543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.583412:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24936448->24940543] (req 24936448->24940543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.583420:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a035 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24936448->24940543] (req 24936448->24940543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.583518:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484349184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 66531us (66878us total) trans 4294973406 rc 0/0 00000100:00100000:3.0:1789760783.583529:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6117 00000100:00100000:3.0:1789760783.583539:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772809606749 time 772847903480 state 0x2 ref 0 00000100:00100000:3.0:1789760783.590892:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.591026:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.591810:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.591850:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.591875:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760783.591985:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484349440 00000100:00100000:0.0:1789760783.591994:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484349440 00000100:00100000:0.0:1789760783.592094:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484349696 00000100:00100000:0.0:1789760783.592098:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484349696 00000100:00100000:0.0:1789760783.592126:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484349568 00000100:00100000:0.0:1789760783.592130:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484349568 00000100:00100000:0.0:1789760783.592143:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484349824 00000100:00100000:0.0:1789760783.592145:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484349824 00000100:00100000:0.0:1789760783.592162:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1001 00000100:00100000:0.0:1789760783.592174:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484349440:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760783.592234:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484349312 00000100:00100000:2.0:1789760783.592251:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484349312 00000100:00100000:0.0:1789760783.592259:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abf800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484349440:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 84us (1376us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760783.592270:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1001 00000100:00100000:0.0:1789760783.592306:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1002 00000100:00100000:0.0:1789760783.592320:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484349696:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760783.592345:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 495 00000100:00100000:0.0:1789760783.592354:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abc700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484349696:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 37us (1325us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760783.592365:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c803ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484349312:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760783.592377:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1002 00000100:00100000:0.0:1789760783.592400:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1003 00000100:00100000:0.0:1789760783.592411:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484349568:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760783.592446:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c803ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484349312:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 89us (638us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760783.592460:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484349568:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 46us (602us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760783.592462:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 495 00000100:00100000:0.0:1789760783.592482:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1003 00000100:00100000:0.0:1789760783.592506:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1004 00000100:00100000:0.0:1789760783.592516:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484349824:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760783.592568:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484349824:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 53us (690us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760783.592577:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1004 00000100:00080000:3.0:1789760783.592784:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:3.0:1789760783.592789:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:1.0:1789760783.593340:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.593407:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484349952 00000100:00100000:2.0:1789760783.593416:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484349952 00000100:00100000:2.0:1789760783.593470:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772809606749 time 772847903480 state 0x0 ref 1 00000100:00100000:2.0:1789760783.593486:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772848163896 time 772847903480 state 0x1 ref 1 00000100:00100000:2.0:1789760783.593519:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772848163896 time 772924450216 state 0x0 ref 1 00000100:00100000:2.0:1789760783.593531:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484349952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.593588:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24940544, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:2.0:1789760783.593598:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.593618:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:2.0:1789760783.593622:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.593630:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24940544->24944639] (req 24940544->24944639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.593647:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24940544->24944639] (req 24940544->24944639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760783.645844:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24940544 -> 24944640 00010000:00010000:0.0:1789760783.645863:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24940544->24944639] (req 24940544->24944639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.645875:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24940544->24944639] (req 24940544->24944639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.645883:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24940544->24944639] (req 24940544->24944639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.645890:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760783.645894:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24940544->24944639] (req 24940544->24944639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.645903:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24940544->24944639] (req 24940544->24944639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.645931:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24940544->24944639] (req 24940544->24944639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.645940:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a03c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24940544->24944639] (req 24940544->24944639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760783.646034:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484349952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52500us (52692us total) trans 4294973407 rc 0/0 00000100:00100000:0.0:1789760783.646045:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6118 00000100:00100000:0.0:1789760783.646055:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772848163896 time 772924450216 state 0x2 ref 0 00000001:00080000:3.0:1789760783.647904:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973407 is committed 00000100:00100000:2.0:1789760783.652997:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.653187:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484350080 00000100:00100000:3.0:1789760783.653203:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484350080 00000100:00100000:3.0:1789760783.653283:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772848163896 time 772924450216 state 0x0 ref 1 00000100:00100000:3.0:1789760783.653302:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772924710632 time 772924450216 state 0x1 ref 1 00000100:00100000:3.0:1789760783.653340:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772924710632 time 772984270826 state 0x0 ref 1 00000100:00100000:3.0:1789760783.653356:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484350080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760783.653424:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24944640, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:3.0:1789760783.653435:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.653456:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=772s v=(0 0) 00010000:00010000:3.0:1789760783.653462:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760783.653469:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24944640->24948735] (req 24944640->24948735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.653487:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24944640->24948735] (req 24944640->24948735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760783.696309:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24944640 -> 24948736 00010000:00010000:3.0:1789760783.696328:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24944640->24948735] (req 24944640->24948735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.696343:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24944640->24948735] (req 24944640->24948735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.696352:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24944640->24948735] (req 24944640->24948735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.696358:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.696363:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24944640->24948735] (req 24944640->24948735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.696372:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24944640->24948735] (req 24944640->24948735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.696395:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24944640->24948735] (req 24944640->24948735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.696403:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a043 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24944640->24948735] (req 24944640->24948735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.696493:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484350080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43135us (43570us total) trans 4294973408 rc 0/0 00000100:00100000:3.0:1789760783.696507:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6119 00000100:00100000:3.0:1789760783.696517:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772924710632 time 772984270826 state 0x2 ref 0 00000001:00080000:0.0:1789760783.697610:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973408 is committed 00000100:00100000:2.0:1789760783.699192:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760783.699270:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484350208 00000100:00100000:0.0:1789760783.699279:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484350208 00000100:00100000:0.0:1789760783.699334:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772924710632 time 772984270826 state 0x0 ref 1 00000100:00100000:0.0:1789760783.699350:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772984531242 time 772984270826 state 0x1 ref 1 00000100:00100000:0.0:1789760783.699377:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772984531242 time 773030308696 state 0x0 ref 1 00000100:00100000:0.0:1789760783.699390:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484350208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760783.699435:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24948736, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:0.0:1789760783.699443:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.699458:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760783.699462:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760783.699469:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24948736->24952831] (req 24948736->24952831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.699482:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24948736->24952831] (req 24948736->24952831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760783.734638:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973409 is committed 00002000:00010000:0.0:1789760783.734739:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24948736 -> 24952832 00010000:00010000:0.0:1789760783.734755:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24948736->24952831] (req 24948736->24952831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.734768:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24948736->24952831] (req 24948736->24952831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.734778:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24948736->24952831] (req 24948736->24952831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.734797:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760783.734801:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24948736->24952831] (req 24948736->24952831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.734810:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24948736->24952831] (req 24948736->24952831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.734831:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24948736->24952831] (req 24948736->24952831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.734838:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a04a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24948736->24952831] (req 24948736->24952831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760783.734932:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484350208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35540us (35743us total) trans 4294973409 rc 0/0 00000100:00100000:0.0:1789760783.734942:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6120 00000100:00100000:0.0:1789760783.734951:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772984531242 time 773030308696 state 0x2 ref 0 00000100:00100000:2.0:1789760783.739062:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760783.739441:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484350336 00000100:00100000:0.0:1789760783.739451:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484350336 00000100:00100000:0.0:1789760783.739513:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 772984531242 time 773030308696 state 0x0 ref 1 00000100:00100000:0.0:1789760783.739527:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773030569112 time 773030308696 state 0x1 ref 1 00000100:00100000:0.0:1789760783.739546:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773030569112 time 773070478377 state 0x0 ref 1 00000100:00100000:0.0:1789760783.739557:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484350336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760783.739602:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24952832, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:0.0:1789760783.739610:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.739624:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760783.739630:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760783.739636:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24952832->24956927] (req 24952832->24956927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.739649:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24952832->24956927] (req 24952832->24956927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760783.766489:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973410 is committed 00002000:00010000:3.0:1789760783.766818:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24952832 -> 24956928 00010000:00010000:3.0:1789760783.766833:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24952832->24956927] (req 24952832->24956927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.766845:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24952832->24956927] (req 24952832->24956927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.766852:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24952832->24956927] (req 24952832->24956927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.766859:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.766863:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24952832->24956927] (req 24952832->24956927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.766871:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24952832->24956927] (req 24952832->24956927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.766888:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24952832->24956927] (req 24952832->24956927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.766895:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a051 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24952832->24956927] (req 24952832->24956927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.766994:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484350336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27434us (27937us total) trans 4294973410 rc 0/0 00000100:00100000:3.0:1789760783.767013:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6121 00000100:00100000:3.0:1789760783.767022:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773030569112 time 773070478377 state 0x2 ref 0 00000100:00100000:2.0:1789760783.770574:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760783.770660:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484350464 00000100:00100000:0.0:1789760783.770670:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484350464 00000100:00100000:0.0:1789760783.770727:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773030569112 time 773070478377 state 0x0 ref 1 00000100:00100000:0.0:1789760783.770743:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773070738793 time 773070478377 state 0x1 ref 1 00000100:00100000:0.0:1789760783.770773:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773070738793 time 773101704174 state 0x0 ref 1 00000100:00100000:0.0:1789760783.770787:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484350464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760783.770842:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24956928, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:0.0:1789760783.770852:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.770867:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760783.770870:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760783.770875:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24956928->24961023] (req 24956928->24961023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.770887:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24956928->24961023] (req 24956928->24961023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760783.804857:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973411 is committed 00002000:00010000:1.0:1789760783.804935:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24956928 -> 24961024 00010000:00010000:1.0:1789760783.804949:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24956928->24961023] (req 24956928->24961023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.804962:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24956928->24961023] (req 24956928->24961023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.804972:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24956928->24961023] (req 24956928->24961023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.804979:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760783.804983:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24956928->24961023] (req 24956928->24961023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.804993:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24956928->24961023] (req 24956928->24961023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.805022:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24956928->24961023] (req 24956928->24961023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.805030:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a058 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24956928->24961023] (req 24956928->24961023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760783.805277:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484350464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34489us (34709us total) trans 4294973411 rc 0/0 00000100:00100000:1.0:1789760783.805287:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6122 00000100:00100000:1.0:1789760783.805297:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773070738793 time 773101704174 state 0x2 ref 0 00000100:00100000:3.0:1789760783.808520:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760783.808622:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484350592 00000100:00100000:0.0:1789760783.808632:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484350592 00000100:00100000:0.0:1789760783.808694:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773070738793 time 773101704174 state 0x0 ref 1 00000100:00100000:0.0:1789760783.808709:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773101964590 time 773101704174 state 0x1 ref 1 00000100:00100000:0.0:1789760783.808738:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773101964590 time 773139669270 state 0x0 ref 1 00000100:00100000:0.0:1789760783.808750:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484350592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760783.808893:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24961024, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:0.0:1789760783.808902:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.808918:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760783.808923:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760783.808930:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24961024->24965119] (req 24961024->24965119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.808943:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24961024->24965119] (req 24961024->24965119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760783.833765:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24961024 -> 24965120 00010000:00010000:2.0:1789760783.833794:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24961024->24965119] (req 24961024->24965119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.833808:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24961024->24965119] (req 24961024->24965119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.833817:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24961024->24965119] (req 24961024->24965119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.833825:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760783.833829:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24961024->24965119] (req 24961024->24965119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.833839:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24961024->24965119] (req 24961024->24965119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.833859:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24961024->24965119] (req 24961024->24965119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.833867:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a05f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24961024->24965119] (req 24961024->24965119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760783.833960:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484350592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25209us (25445us total) trans 4294973412 rc 0/0 00000100:00100000:2.0:1789760783.833971:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6123 00000100:00100000:2.0:1789760783.833981:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773101964590 time 773139669270 state 0x2 ref 0 00000001:00080000:3.0:1789760783.834169:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973412 is committed 00000100:00100000:3.0:1789760783.838920:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.839022:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484350720 00000100:00100000:2.0:1789760783.839033:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484350720 00000100:00100000:2.0:1789760783.839251:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773101964590 time 773139669270 state 0x0 ref 1 00000100:00100000:2.0:1789760783.839267:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773139929686 time 773139669270 state 0x1 ref 1 00000100:00100000:2.0:1789760783.839297:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773139929686 time 773170227399 state 0x0 ref 1 00000100:00100000:2.0:1789760783.839310:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7069850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484350720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.839356:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24965120, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:2.0:1789760783.839365:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.839380:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:2.0:1789760783.839385:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.839391:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24965120->24969215] (req 24965120->24969215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.839404:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24965120->24969215] (req 24965120->24969215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760783.906493:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24965120 -> 24969216 00010000:00010000:2.0:1789760783.906513:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24965120->24969215] (req 24965120->24969215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.906527:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24965120->24969215] (req 24965120->24969215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.906536:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24965120->24969215] (req 24965120->24969215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.906544:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760783.906549:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24965120->24969215] (req 24965120->24969215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.906558:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24965120->24969215] (req 24965120->24969215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.906588:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24965120->24969215] (req 24965120->24969215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.906597:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a066 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24965120->24969215] (req 24965120->24969215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760783.906691:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7069850 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484350720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67379us (67775us total) trans 4294973413 rc 0/0 00000100:00100000:2.0:1789760783.906703:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6124 00000100:00100000:2.0:1789760783.906713:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773139929686 time 773170227399 state 0x2 ref 0 00000001:00080000:3.0:1789760783.907120:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973413 is committed 00000100:00100000:1.0:1789760783.914273:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760783.914410:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484350848 00000100:00100000:3.0:1789760783.914420:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484350848 00000100:00100000:3.0:1789760783.914473:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773139929686 time 773170227399 state 0x0 ref 1 00000100:00100000:3.0:1789760783.914488:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773170487815 time 773170227399 state 0x1 ref 1 00000100:00100000:3.0:1789760783.914517:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773170487815 time 773245447840 state 0x0 ref 1 00000100:00100000:3.0:1789760783.914529:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484350848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760783.914575:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24969216, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:3.0:1789760783.914584:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.914600:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:3.0:1789760783.914605:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760783.914612:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24969216->24973311] (req 24969216->24973311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.914626:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24969216->24973311] (req 24969216->24973311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760783.940969:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24969216 -> 24973312 00010000:00010000:3.0:1789760783.940988:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24969216->24973311] (req 24969216->24973311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.941013:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24969216->24973311] (req 24969216->24973311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.941023:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24969216->24973311] (req 24969216->24973311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.941031:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760783.941036:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24969216->24973311] (req 24969216->24973311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.941045:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24969216->24973311] (req 24969216->24973311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.941200:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24969216->24973311] (req 24969216->24973311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760783.941210:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a06d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24969216->24973311] (req 24969216->24973311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760783.941341:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484350848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26809us (27076us total) trans 4294973414 rc 0/0 00000100:00100000:3.0:1789760783.941352:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6125 00000100:00100000:3.0:1789760783.941361:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773170487815 time 773245447840 state 0x2 ref 0 00000001:00080000:3.0:1789760783.941398:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973414 is committed 00000100:00100000:1.0:1789760783.945384:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760783.945531:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484350976 00000100:00100000:0.0:1789760783.945541:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484350976 00000100:00100000:0.0:1789760783.945599:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773170487815 time 773245447840 state 0x0 ref 1 00000100:00100000:0.0:1789760783.945614:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773245708256 time 773245447840 state 0x1 ref 1 00000100:00100000:0.0:1789760783.945645:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773245708256 time 773276575920 state 0x0 ref 1 00000100:00100000:0.0:1789760783.945657:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c13b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484350976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760783.945698:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24973312, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:0.0:1789760783.945704:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.945717:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760783.945720:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760783.945724:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24973312->24977407] (req 24973312->24977407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760783.945734:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24973312->24977407] (req 24973312->24977407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760783.981029:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973415 is committed 00002000:00010000:1.0:1789760783.981450:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24973312 -> 24977408 00010000:00010000:1.0:1789760783.981466:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24973312->24977407] (req 24973312->24977407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.981478:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24973312->24977407] (req 24973312->24977407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.981487:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24973312->24977407] (req 24973312->24977407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.981494:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760783.981499:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24973312->24977407] (req 24973312->24977407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.981509:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24973312->24977407] (req 24973312->24977407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.981529:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24973312->24977407] (req 24973312->24977407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760783.981537:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a074 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24973312->24977407] (req 24973312->24977407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760783.981635:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c13b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484350976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35975us (36256us total) trans 4294973415 rc 0/0 00000100:00100000:1.0:1789760783.981646:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6126 00000100:00100000:1.0:1789760783.981656:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773245708256 time 773276575920 state 0x2 ref 0 00000100:00100000:3.0:1789760783.985516:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760783.985602:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484351104 00000100:00100000:2.0:1789760783.985613:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484351104 00000100:00100000:2.0:1789760783.985663:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773245708256 time 773276575920 state 0x0 ref 1 00000100:00100000:2.0:1789760783.985679:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773276836336 time 773276575920 state 0x1 ref 1 00000100:00100000:2.0:1789760783.985733:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773276836336 time 773316664156 state 0x0 ref 1 00000100:00100000:2.0:1789760783.985747:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484351104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760783.985804:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24977408, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:2.0:1789760783.985813:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.985827:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:2.0:1789760783.985832:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760783.985838:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24977408->24981503] (req 24977408->24981503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760783.985851:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24977408->24981503] (req 24977408->24981503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760784.027283:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973416 is committed 00002000:00010000:2.0:1789760784.027543:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24977408 -> 24981504 00010000:00010000:2.0:1789760784.027559:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24977408->24981503] (req 24977408->24981503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.027572:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24977408->24981503] (req 24977408->24981503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.027587:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24977408->24981503] (req 24977408->24981503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.027662:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760784.027668:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24977408->24981503] (req 24977408->24981503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.027679:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24977408->24981503] (req 24977408->24981503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.027698:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24977408->24981503] (req 24977408->24981503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.027708:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a07b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24977408->24981503] (req 24977408->24981503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760784.027849:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484351104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42099us (42336us total) trans 4294973416 rc 0/0 00000100:00100000:2.0:1789760784.027859:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6127 00000100:00100000:2.0:1789760784.027868:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773276836336 time 773316664156 state 0x2 ref 0 00000100:00100000:2.0:1789760784.032380:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760784.032756:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484351232 00000100:00100000:0.0:1789760784.032766:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484351232 00000100:00080000:3.0:1789760784.032901:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760784.032902:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773276836336 time 773316664156 state 0x0 ref 1 00000100:00100000:0.0:1789760784.032919:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773316924572 time 773316664156 state 0x1 ref 1 00000100:00100000:0.0:1789760784.032952:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773316924572 time 773363882739 state 0x0 ref 1 00000100:00100000:0.0:1789760784.032969:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484351232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760784.033026:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24981504, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:0.0:1789760784.033035:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.033051:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760784.033055:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760784.033179:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24981504->24985599] (req 24981504->24985599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.033197:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24981504->24985599] (req 24981504->24985599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760784.073023:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973417 is committed 00002000:00010000:2.0:1789760784.073202:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24981504 -> 24985600 00010000:00010000:2.0:1789760784.073216:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24981504->24985599] (req 24981504->24985599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.073236:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24981504->24985599] (req 24981504->24985599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.073245:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24981504->24985599] (req 24981504->24985599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.073252:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760784.073257:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24981504->24985599] (req 24981504->24985599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.073265:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24981504->24985599] (req 24981504->24985599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.073286:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24981504->24985599] (req 24981504->24985599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.073294:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a082 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24981504->24985599] (req 24981504->24985599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760784.073391:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484351232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40424us (41015us total) trans 4294973417 rc 0/0 00000100:00100000:2.0:1789760784.073401:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6128 00000100:00100000:2.0:1789760784.073410:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773316924572 time 773363882739 state 0x2 ref 0 00000100:00100000:2.0:1789760784.077546:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760784.077627:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484351360 00000100:00100000:3.0:1789760784.077637:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484351360 00000100:00100000:3.0:1789760784.077688:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773316924572 time 773363882739 state 0x0 ref 1 00000100:00100000:3.0:1789760784.077703:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773364143155 time 773363882739 state 0x1 ref 1 00000100:00100000:3.0:1789760784.077731:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773364143155 time 773408662125 state 0x0 ref 1 00000100:00100000:3.0:1789760784.077743:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484351360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760784.077799:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24985600, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:3.0:1789760784.077808:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.077824:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:3.0:1789760784.077830:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760784.077836:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24985600->24989695] (req 24985600->24989695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.077850:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24985600->24989695] (req 24985600->24989695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760784.123672:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973418 is committed 00002000:00010000:1.0:1789760784.123800:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24985600 -> 24989696 00010000:00010000:1.0:1789760784.123815:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24985600->24989695] (req 24985600->24989695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.123827:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24985600->24989695] (req 24985600->24989695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.123836:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24985600->24989695] (req 24985600->24989695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.123843:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760784.123847:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24985600->24989695] (req 24985600->24989695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.123855:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24985600->24989695] (req 24985600->24989695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.123872:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24985600->24989695] (req 24985600->24989695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.123879:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a089 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24985600->24989695] (req 24985600->24989695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760784.123970:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484351360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46224us (46430us total) trans 4294973418 rc 0/0 00000100:00100000:1.0:1789760784.123983:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6129 00000100:00100000:1.0:1789760784.123993:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773364143155 time 773408662125 state 0x2 ref 0 00000100:00100000:1.0:1789760784.136412:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760784.137033:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484351488 00000100:00100000:0.0:1789760784.137046:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484351488 00000100:00100000:0.0:1789760784.137655:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773364143155 time 773408662125 state 0x0 ref 1 00000100:00100000:0.0:1789760784.137675:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773408922541 time 773408662125 state 0x1 ref 1 00000100:00100000:2.0:1789760784.137836:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773408922541 time 773468763041 state 0x0 ref 1 00000100:00100000:2.0:1789760784.137853:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484351488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760784.137904:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24989696, mtime: 0x6aad950f, atime: 0x0, ctime: 0x6aad950f, blocks: 0xc007 00010000:00010000:2.0:1789760784.137913:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.137928:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:2.0:1789760784.137933:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760784.137939:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24989696->24993791] (req 24989696->24993791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.137954:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24989696->24993791] (req 24989696->24993791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760784.187130:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24989696 -> 24993792 00002000:00010000:2.0:1789760784.187141:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760783 -> 1789760784 00002000:00010000:2.0:1789760784.187144:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760783 -> 1789760784 00010000:00010000:2.0:1789760784.187158:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24989696->24993791] (req 24989696->24993791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.187172:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24989696->24993791] (req 24989696->24993791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.187180:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24989696->24993791] (req 24989696->24993791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.187316:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760784.187322:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24989696->24993791] (req 24989696->24993791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.187333:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24989696->24993791] (req 24989696->24993791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.187354:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24989696->24993791] (req 24989696->24993791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.187362:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a090 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24989696->24993791] (req 24989696->24993791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760784.187461:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484351488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49608us (51053us total) trans 4294973419 rc 0/0 00000100:00100000:2.0:1789760784.187472:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6130 00000100:00100000:2.0:1789760784.187481:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773408922541 time 773468763041 state 0x2 ref 0 00000001:00080000:1.0:1789760784.188690:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973419 is committed 00000100:00100000:2.0:1789760784.193776:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760784.193852:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484351616 00000100:00100000:0.0:1789760784.193861:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484351616 00000100:00100000:0.0:1789760784.193920:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773408922541 time 773468763041 state 0x0 ref 1 00000100:00100000:0.0:1789760784.193935:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773469023457 time 773468763041 state 0x1 ref 1 00000100:00100000:0.0:1789760784.193967:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773469023457 time 773524897831 state 0x0 ref 1 00000100:00100000:0.0:1789760784.193981:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484351616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760784.194042:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24993792, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:0.0:1789760784.194054:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.194168:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760784.194174:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760784.194180:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24993792->24997887] (req 24993792->24997887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.194258:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24993792->24997887] (req 24993792->24997887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760784.241931:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973420 is committed 00002000:00010000:0.0:1789760784.242577:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24993792 -> 24997888 00010000:00010000:0.0:1789760784.242663:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24993792->24997887] (req 24993792->24997887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.242677:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24993792->24997887] (req 24993792->24997887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.242688:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24993792->24997887] (req 24993792->24997887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.242695:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760784.242700:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24993792->24997887] (req 24993792->24997887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.242709:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24993792->24997887] (req 24993792->24997887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.243168:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24993792->24997887] (req 24993792->24997887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.243178:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a097 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24993792->24997887] (req 24993792->24997887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760784.243332:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484351616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49350us (49559us total) trans 4294973420 rc 0/0 00000100:00100000:0.0:1789760784.243343:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6131 00000100:00100000:0.0:1789760784.243352:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773469023457 time 773524897831 state 0x2 ref 0 00000100:00100000:0.0:1789760784.245940:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760784.246172:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484351744 00000100:00100000:2.0:1789760784.246182:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484351744 00000100:00100000:2.0:1789760784.246237:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773469023457 time 773524897831 state 0x0 ref 1 00000100:00100000:2.0:1789760784.246252:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773525158247 time 773524897831 state 0x1 ref 1 00000100:00100000:2.0:1789760784.246717:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773525158247 time 773577644928 state 0x0 ref 1 00000100:00100000:2.0:1789760784.246883:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484351744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760784.246933:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 24997888, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:2.0:1789760784.246942:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.246958:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:2.0:1789760784.246963:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760784.246969:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24997888->25001983] (req 24997888->25001983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.246982:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24997888->25001983] (req 24997888->25001983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.268423:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760784.268627:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c880c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508940928:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760784.268695:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760784.268957:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98dca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508941056:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760784.268985:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760784.269038:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508940928 00000100:00100000:1.0:1789760784.269226:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508941056 00000100:00100000:1.0:1789760784.269231:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508941056 00000100:00080000:0.0:1789760784.269263:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760784.269490:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508940928 00000100:00100000:1.0:1789760784.269512:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 414 00000100:00100000:1.0:1789760784.269522:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98df100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508941056:12345-0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760784.269525:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 415 00000100:00100000:3.0:1789760784.269530:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508940928:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:1.0:1789760784.269544:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527630336 pending 1703936 free 3833593856 avail 3818913792 00002000:00000024:3.0:1789760784.269544:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 527630336 pending 1703936 free 3833593856 avail 3818913792 00000100:00100000:1.0:1789760784.269582:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98df100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508941056:12345-0@lo:13:osp-pre-0-0.0 Request processed in 62us (596us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760784.269583:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508940928:12345-0@lo:13:osp-pre-0-1.0 Request processed in 53us (890us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760784.269589:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 414 00000100:00080000:2.0:1789760784.269620:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1789760784.269679:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c11c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508941184:0@lo:41:osp-pre-0-1.0 00000100:00100000:1.0:1789760784.269709:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760784.269971:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508941184 00000100:00100000:1.0:1789760784.269977:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508941184 00000100:00100000:1.0:1789760784.270021:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 473 00000100:00100000:1.0:1789760784.270028:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508941184:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:1.0:1789760784.270057:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:2.0:1789760784.271735:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c880c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508940928:0@lo:13:osp-pre-0-1.0 00000100:00100000:0.0:1789760784.271893:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98dca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508941056:0@lo:13:osp-pre-0-0.0 00000100:00100000:1.0:1789760784.272061:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508941184:12345-0@lo:41:osp-pre-0-1.0 Request processed in 2032us (2349us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760784.272093:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 473 00000100:00100000:0.0:1789760784.272233:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c11c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508941184:0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760784.273012:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 415 00000001:00080000:2.0:1789760784.290445:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973421 is committed 00002000:00010000:0.0:1789760784.290873:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 24997888 -> 25001984 00010000:00010000:0.0:1789760784.290889:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24997888->25001983] (req 24997888->25001983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.290902:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24997888->25001983] (req 24997888->25001983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.290916:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24997888->25001983] (req 24997888->25001983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.290923:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760784.290929:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24997888->25001983] (req 24997888->25001983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.290937:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24997888->25001983] (req 24997888->25001983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.290957:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24997888->25001983] (req 24997888->25001983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.290965:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a09e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [24997888->25001983] (req 24997888->25001983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760784.291176:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484351744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44292us (45238us total) trans 4294973421 rc 0/0 00000100:00100000:0.0:1789760784.291186:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6132 00000100:00100000:0.0:1789760784.291196:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773525158247 time 773577644928 state 0x2 ref 0 00000100:00100000:1.0:1789760784.294559:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760784.294699:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484351872 00000100:00100000:3.0:1789760784.294709:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484351872 00000100:00100000:3.0:1789760784.294774:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773525158247 time 773577644928 state 0x0 ref 1 00000100:00100000:3.0:1789760784.294797:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773577905344 time 773577644928 state 0x1 ref 1 00000100:00100000:3.0:1789760784.294823:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773577905344 time 773625754279 state 0x0 ref 1 00000100:00100000:3.0:1789760784.294837:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484351872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760784.294884:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25001984, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:3.0:1789760784.294899:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.294942:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:3.0:1789760784.294947:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760784.294954:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25001984->25006079] (req 25001984->25006079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.294967:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25001984->25006079] (req 25001984->25006079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760784.335938:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973422 is committed 00002000:00010000:2.0:1789760784.336400:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25001984 -> 25006080 00010000:00010000:2.0:1789760784.336415:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25001984->25006079] (req 25001984->25006079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.336428:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25001984->25006079] (req 25001984->25006079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.336437:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25001984->25006079] (req 25001984->25006079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.336445:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760784.336450:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25001984->25006079] (req 25001984->25006079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.336459:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25001984->25006079] (req 25001984->25006079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.336479:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25001984->25006079] (req 25001984->25006079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.336487:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a0a5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25001984->25006079] (req 25001984->25006079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760784.336584:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484351872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41745us (42033us total) trans 4294973422 rc 0/0 00000100:00100000:2.0:1789760784.336594:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6133 00000100:00100000:2.0:1789760784.336603:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773577905344 time 773625754279 state 0x2 ref 0 00000100:00100000:2.0:1789760784.339506:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760784.339594:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484352000 00000100:00100000:0.0:1789760784.339611:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484352000 00000100:00100000:0.0:1789760784.339672:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773577905344 time 773625754279 state 0x0 ref 1 00000100:00100000:0.0:1789760784.339687:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773626014695 time 773625754279 state 0x1 ref 1 00000100:00100000:0.0:1789760784.339728:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773626014695 time 773670650727 state 0x0 ref 1 00000100:00100000:0.0:1789760784.339742:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484352000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760784.339800:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25006080, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:0.0:1789760784.339810:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.339826:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760784.339831:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760784.339837:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25006080->25010175] (req 25006080->25010175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.339851:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25006080->25010175] (req 25006080->25010175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760784.391187:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973423 is committed 00002000:00010000:2.0:1789760784.393240:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25006080 -> 25010176 00010000:00010000:2.0:1789760784.393262:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25006080->25010175] (req 25006080->25010175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.393276:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25006080->25010175] (req 25006080->25010175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.393290:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25006080->25010175] (req 25006080->25010175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.393297:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760784.393302:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25006080->25010175] (req 25006080->25010175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.393311:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25006080->25010175] (req 25006080->25010175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.393332:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25006080->25010175] (req 25006080->25010175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.393340:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a0ac lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25006080->25010175] (req 25006080->25010175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760784.393455:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484352000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53711us (53955us total) trans 4294973423 rc 0/0 00000100:00100000:2.0:1789760784.393467:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6134 00000100:00100000:2.0:1789760784.393477:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773626014695 time 773670650727 state 0x2 ref 0 00000100:00100000:1.0:1789760784.396649:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760784.397009:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484352128 00000100:00100000:0.0:1789760784.397020:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484352128 00000100:00100000:0.0:1789760784.397095:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773626014695 time 773670650727 state 0x0 ref 1 00000100:00100000:0.0:1789760784.397111:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773670911143 time 773670650727 state 0x1 ref 1 00000100:00100000:0.0:1789760784.397185:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773670911143 time 773728115241 state 0x0 ref 1 00000100:00100000:0.0:1789760784.397201:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484352128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760784.397262:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25010176, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:0.0:1789760784.397271:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.397289:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760784.397293:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760784.397298:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25010176->25014271] (req 25010176->25014271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.397312:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25010176->25014271] (req 25010176->25014271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760784.439472:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973424 is committed 00002000:00010000:0.0:1789760784.439627:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25010176 -> 25014272 00010000:00010000:0.0:1789760784.439643:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25010176->25014271] (req 25010176->25014271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.439657:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25010176->25014271] (req 25010176->25014271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.439665:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25010176->25014271] (req 25010176->25014271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.439672:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760784.439676:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25010176->25014271] (req 25010176->25014271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.439684:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25010176->25014271] (req 25010176->25014271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.439708:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25010176->25014271] (req 25010176->25014271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.439716:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a0b3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25010176->25014271] (req 25010176->25014271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760784.439828:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484352128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42625us (43182us total) trans 4294973424 rc 0/0 00000100:00100000:0.0:1789760784.439839:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6135 00000100:00100000:0.0:1789760784.439850:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773670911143 time 773728115241 state 0x2 ref 0 00000100:00100000:1.0:1789760784.444422:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760784.445569:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484352256 00000100:00100000:2.0:1789760784.445654:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484352256 00000100:00100000:3.0:1789760784.445827:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773670911143 time 773728115241 state 0x0 ref 1 00000100:00100000:3.0:1789760784.445844:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773728375657 time 773728115241 state 0x1 ref 1 00000100:00100000:3.0:1789760784.445879:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773728375657 time 773776809789 state 0x0 ref 1 00000100:00100000:3.0:1789760784.445893:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484352256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760784.445946:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25014272, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:3.0:1789760784.445954:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.445970:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:3.0:1789760784.445974:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760784.445981:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25014272->25018367] (req 25014272->25018367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.446009:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25014272->25018367] (req 25014272->25018367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760784.480527:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25014272 -> 25018368 00010000:00010000:0.0:1789760784.480548:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25014272->25018367] (req 25014272->25018367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.480562:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25014272->25018367] (req 25014272->25018367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.480572:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25014272->25018367] (req 25014272->25018367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.480579:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760784.480583:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25014272->25018367] (req 25014272->25018367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.480598:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25014272->25018367] (req 25014272->25018367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.480619:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25014272->25018367] (req 25014272->25018367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.480627:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a0ba lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25014272->25018367] (req 25014272->25018367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760784.480734:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484352256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34838us (36316us total) trans 4294973425 rc 0/0 00000100:00100000:0.0:1789760784.480745:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6136 00000100:00100000:0.0:1789760784.480755:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773728375657 time 773776809789 state 0x2 ref 0 00000001:00080000:1.0:1789760784.480976:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973425 is committed 00000100:00100000:3.0:1789760784.486327:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760784.486398:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484352384 00000100:00100000:2.0:1789760784.486410:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484352384 00000100:00100000:2.0:1789760784.486468:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773728375657 time 773776809789 state 0x0 ref 1 00000100:00100000:2.0:1789760784.486487:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773777070205 time 773776809789 state 0x1 ref 1 00000100:00100000:2.0:1789760784.486517:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773777070205 time 773817448213 state 0x0 ref 1 00000100:00100000:2.0:1789760784.486530:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484352384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760784.486584:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25018368, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:2.0:1789760784.486594:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.486614:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:2.0:1789760784.486618:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760784.486624:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25018368->25022463] (req 25018368->25022463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.486638:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25018368->25022463] (req 25018368->25022463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760784.532389:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973426 is committed 00002000:00010000:3.0:1789760784.532483:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25018368 -> 25022464 00010000:00010000:3.0:1789760784.532500:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25018368->25022463] (req 25018368->25022463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.532514:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25018368->25022463] (req 25018368->25022463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.532523:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25018368->25022463] (req 25018368->25022463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.532530:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760784.532535:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25018368->25022463] (req 25018368->25022463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.532544:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25018368->25022463] (req 25018368->25022463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.532564:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25018368->25022463] (req 25018368->25022463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.532571:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0c1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25018368->25022463] (req 25018368->25022463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760784.534929:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484352384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48388us (48600us total) trans 4294973426 rc 0/0 00000100:00100000:3.0:1789760784.534944:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6137 00000100:00100000:3.0:1789760784.534954:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773777070205 time 773817448213 state 0x2 ref 0 00000100:00100000:1.0:1789760784.538902:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760784.539006:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484352512 00000100:00100000:2.0:1789760784.539015:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484352512 00000100:00100000:2.0:1789760784.539302:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773777070205 time 773817448213 state 0x0 ref 1 00000100:00100000:2.0:1789760784.539318:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773817708629 time 773817448213 state 0x1 ref 1 00000100:00100000:2.0:1789760784.539386:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773817708629 time 773870315642 state 0x0 ref 1 00000100:00100000:2.0:1789760784.539400:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484352512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760784.539514:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25022464, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:2.0:1789760784.539523:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.539541:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:2.0:1789760784.539545:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760784.539552:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25022464->25026559] (req 25022464->25026559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.539566:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25022464->25026559] (req 25022464->25026559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760784.568587:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973427 is committed 00002000:00010000:2.0:1789760784.568731:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25022464 -> 25026560 00010000:00010000:2.0:1789760784.568746:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25022464->25026559] (req 25022464->25026559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.568758:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25022464->25026559] (req 25022464->25026559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.568768:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25022464->25026559] (req 25022464->25026559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.568786:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760784.568791:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25022464->25026559] (req 25022464->25026559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.568799:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25022464->25026559] (req 25022464->25026559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.568820:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25022464->25026559] (req 25022464->25026559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.568828:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0c8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25022464->25026559] (req 25022464->25026559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760784.568917:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484352512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29516us (30022us total) trans 4294973427 rc 0/0 00000100:00100000:2.0:1789760784.568928:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6138 00000100:00100000:2.0:1789760784.568938:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773817708629 time 773870315642 state 0x2 ref 0 00000100:00100000:1.0:1789760784.573015:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760784.573214:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484352640 00000100:00100000:2.0:1789760784.573224:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484352640 00000100:00100000:2.0:1789760784.573277:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773817708629 time 773870315642 state 0x0 ref 1 00000100:00100000:2.0:1789760784.573292:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773870576058 time 773870315642 state 0x1 ref 1 00000100:00100000:2.0:1789760784.573321:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773870576058 time 773904252256 state 0x0 ref 1 00000100:00100000:2.0:1789760784.573334:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484352640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760784.573382:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25026560, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:2.0:1789760784.573392:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.573408:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:2.0:1789760784.573412:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760784.573418:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25026560->25030655] (req 25026560->25030655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.573432:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25026560->25030655] (req 25026560->25030655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760784.603305:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973428 is committed 00002000:00010000:2.0:1789760784.603438:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25026560 -> 25030656 00010000:00010000:2.0:1789760784.603453:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25026560->25030655] (req 25026560->25030655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.603466:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25026560->25030655] (req 25026560->25030655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.603475:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25026560->25030655] (req 25026560->25030655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.603483:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760784.603488:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25026560->25030655] (req 25026560->25030655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.603497:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25026560->25030655] (req 25026560->25030655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.603517:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25026560->25030655] (req 25026560->25030655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.603523:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a0cf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25026560->25030655] (req 25026560->25030655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760784.603621:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484352640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30284us (30609us total) trans 4294973428 rc 0/0 00000100:00100000:2.0:1789760784.603631:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6139 00000100:00100000:2.0:1789760784.603641:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773870576058 time 773904252256 state 0x2 ref 0 00000100:00100000:2.0:1789760784.606949:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760784.607301:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484352768 00000100:00100000:0.0:1789760784.607311:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484352768 00000100:00100000:0.0:1789760784.607379:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773870576058 time 773904252256 state 0x0 ref 1 00000100:00100000:0.0:1789760784.607395:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773904512672 time 773904252256 state 0x1 ref 1 00000100:00100000:0.0:1789760784.607460:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773904512672 time 773938390299 state 0x0 ref 1 00000100:00100000:0.0:1789760784.607475:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484352768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760784.607526:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25030656, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:0.0:1789760784.607535:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.607551:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:0.0:1789760784.607556:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760784.607562:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25030656->25034751] (req 25030656->25034751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.607577:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25030656->25034751] (req 25030656->25034751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760784.634332:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973429 is committed 00002000:00010000:1.0:1789760784.634419:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25030656 -> 25034752 00010000:00010000:1.0:1789760784.634431:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25030656->25034751] (req 25030656->25034751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.634450:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25030656->25034751] (req 25030656->25034751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.634457:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25030656->25034751] (req 25030656->25034751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.634462:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760784.634465:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25030656->25034751] (req 25030656->25034751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.634471:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25030656->25034751] (req 25030656->25034751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.634485:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25030656->25034751] (req 25030656->25034751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.634490:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a0d6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25030656->25034751] (req 25030656->25034751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760784.634629:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484352768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27154us (27687us total) trans 4294973429 rc 0/0 00000100:00100000:1.0:1789760784.634637:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6140 00000100:00100000:1.0:1789760784.634645:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773904512672 time 773938390299 state 0x2 ref 0 00000100:00100000:2.0:1789760784.639698:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760784.639760:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484352896 00000100:00100000:3.0:1789760784.639770:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484352896 00000100:00100000:3.0:1789760784.639888:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773904512672 time 773938390299 state 0x0 ref 1 00000100:00100000:3.0:1789760784.639903:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773938650715 time 773938390299 state 0x1 ref 1 00000100:00100000:3.0:1789760784.640289:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773938650715 time 773971217577 state 0x0 ref 1 00000100:00100000:3.0:1789760784.640304:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484352896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760784.640351:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25034752, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:3.0:1789760784.640359:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.640375:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=773s v=(0 0) 00010000:00010000:3.0:1789760784.640379:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760784.640385:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25034752->25038847] (req 25034752->25038847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.640399:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25034752->25038847] (req 25034752->25038847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760784.678106:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25034752 -> 25038848 00010000:00010000:3.0:1789760784.678126:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25034752->25038847] (req 25034752->25038847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.678139:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25034752->25038847] (req 25034752->25038847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.678149:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25034752->25038847] (req 25034752->25038847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.678155:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760784.678160:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25034752->25038847] (req 25034752->25038847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.678168:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25034752->25038847] (req 25034752->25038847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.678189:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25034752->25038847] (req 25034752->25038847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.678196:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0dd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25034752->25038847] (req 25034752->25038847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760784.678293:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484352896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37988us (38596us total) trans 4294973430 rc 0/0 00000100:00100000:3.0:1789760784.678307:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6141 00000100:00100000:3.0:1789760784.678315:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773938650715 time 773971217577 state 0x2 ref 0 00000001:00080000:0.0:1789760784.678781:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973430 is committed 00000100:00100000:2.0:1789760784.681345:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760784.681443:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484353024 00000100:00100000:0.0:1789760784.681453:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484353024 00000100:00100000:0.0:1789760784.681583:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773938650715 time 773971217577 state 0x0 ref 1 00000100:00100000:0.0:1789760784.681598:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773971477993 time 773971217577 state 0x1 ref 1 00000100:00100000:0.0:1789760784.681675:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773971477993 time 774012605496 state 0x0 ref 1 00000100:00100000:0.0:1789760784.681690:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484353024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760784.681740:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25038848, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:0.0:1789760784.681749:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.681771:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:0.0:1789760784.681785:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760784.681791:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25038848->25042943] (req 25038848->25042943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.681807:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25038848->25042943] (req 25038848->25042943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760784.716694:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25038848 -> 25042944 00010000:00010000:0.0:1789760784.716714:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25038848->25042943] (req 25038848->25042943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.716728:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25038848->25042943] (req 25038848->25042943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.716737:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25038848->25042943] (req 25038848->25042943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.716745:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760784.716749:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25038848->25042943] (req 25038848->25042943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.716757:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25038848->25042943] (req 25038848->25042943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.716787:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25038848->25042943] (req 25038848->25042943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.716796:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0e4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25038848->25042943] (req 25038848->25042943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760784.716890:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484353024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35200us (35550us total) trans 4294973431 rc 0/0 00000100:00100000:0.0:1789760784.716901:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6142 00000100:00100000:0.0:1789760784.716910:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773971477993 time 774012605496 state 0x2 ref 0 00000001:00080000:0.0:1789760784.716950:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973431 is committed 00000100:00100000:1.0:1789760784.721522:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760784.721620:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484353152 00000100:00100000:2.0:1789760784.721635:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484353152 00000100:00100000:2.0:1789760784.721694:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 773971477993 time 774012605496 state 0x0 ref 1 00000100:00100000:2.0:1789760784.721710:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774012865912 time 774012605496 state 0x1 ref 1 00000100:00100000:2.0:1789760784.721741:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774012865912 time 774052672606 state 0x0 ref 1 00000100:00100000:2.0:1789760784.721761:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484353152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760784.721816:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25042944, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:2.0:1789760784.721825:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.721842:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:2.0:1789760784.721847:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760784.721853:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25042944->25047039] (req 25042944->25047039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.721873:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25042944->25047039] (req 25042944->25047039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760784.768351:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973432 is committed 00002000:00010000:0.0:1789760784.768481:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25042944 -> 25047040 00010000:00010000:0.0:1789760784.768502:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25042944->25047039] (req 25042944->25047039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.768519:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25042944->25047039] (req 25042944->25047039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.768529:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25042944->25047039] (req 25042944->25047039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.768536:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760784.768542:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25042944->25047039] (req 25042944->25047039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.768551:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25042944->25047039] (req 25042944->25047039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.768582:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25042944->25047039] (req 25042944->25047039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.768591:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a0eb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25042944->25047039] (req 25042944->25047039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760784.768758:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484353152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46997us (47243us total) trans 4294973432 rc 0/0 00000100:00100000:0.0:1789760784.768770:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6143 00000100:00100000:0.0:1789760784.768792:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774012865912 time 774052672606 state 0x2 ref 0 00000100:00100000:1.0:1789760784.772917:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760784.773050:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484353280 00000100:00100000:0.0:1789760784.773846:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484353280 00000100:00100000:0.0:1789760784.773949:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774012865912 time 774052672606 state 0x0 ref 1 00000100:00100000:0.0:1789760784.773969:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774052933022 time 774052672606 state 0x1 ref 1 00000100:00100000:0.0:1789760784.774138:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774052933022 time 774105067076 state 0x0 ref 1 00000100:00100000:0.0:1789760784.774163:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484353280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760784.774303:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25047040, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:0.0:1789760784.774316:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.774337:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:0.0:1789760784.774343:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760784.774350:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25047040->25051135] (req 25047040->25051135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.774367:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25047040->25051135] (req 25047040->25051135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760784.781384:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:1.0:1789760784.781419:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c3e86680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508941312:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760784.781528:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760784.781666:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508941312 00000100:00100000:0.0:1789760784.781676:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508941312 00000100:00100000:0.0:1789760784.781740:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 474 00000100:00100000:0.0:1789760784.781751:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508941312:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:0.0:1789760784.781803:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:0.0:1789760784.781862:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508941312:12345-0@lo:41:osp-pre-1-0.0 Request processed in 113us (334us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760784.781871:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 474 00000100:00100000:0.0:1789760784.781927:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c3e86680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508941312:0@lo:41:osp-pre-1-0.0 00002000:00010000:1.0:1789760784.812830:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25047040 -> 25051136 00010000:00010000:1.0:1789760784.812859:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25047040->25051135] (req 25047040->25051135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.812874:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25047040->25051135] (req 25047040->25051135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.812884:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25047040->25051135] (req 25047040->25051135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.812892:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760784.812898:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25047040->25051135] (req 25047040->25051135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.812915:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25047040->25051135] (req 25047040->25051135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.812942:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25047040->25051135] (req 25047040->25051135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.812952:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a0f2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25047040->25051135] (req 25047040->25051135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760784.813134:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e85c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484353280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38978us (40225us total) trans 4294973433 rc 0/0 00000100:00100000:1.0:1789760784.813148:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6144 00000100:00100000:1.0:1789760784.813161:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774052933022 time 774105067076 state 0x2 ref 0 00000001:00080000:1.0:1789760784.813222:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973433 is committed 00000100:00100000:2.0:1789760784.816698:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760784.817000:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484353408 00000100:00100000:3.0:1789760784.817014:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484353408 00000100:00100000:3.0:1789760784.817137:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774052933022 time 774105067076 state 0x0 ref 1 00000100:00100000:3.0:1789760784.817155:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774105327492 time 774105067076 state 0x1 ref 1 00000100:00100000:3.0:1789760784.817181:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774105327492 time 774148111928 state 0x0 ref 1 00000100:00100000:3.0:1789760784.817196:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484353408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760784.817436:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25051136, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:3.0:1789760784.817447:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.817466:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:3.0:1789760784.817471:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760784.817478:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25051136->25055231] (req 25051136->25055231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760784.817494:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25051136->25055231] (req 25051136->25055231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760784.884491:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25051136 -> 25055232 00010000:00010000:2.0:1789760784.884528:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25051136->25055231] (req 25051136->25055231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.884543:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25051136->25055231] (req 25051136->25055231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.884558:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25051136->25055231] (req 25051136->25055231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.884566:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760784.884571:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25051136->25055231] (req 25051136->25055231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.884580:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25051136->25055231] (req 25051136->25055231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.884605:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25051136->25055231] (req 25051136->25055231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.884613:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a0f9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25051136->25055231] (req 25051136->25055231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760784.884735:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484353408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67536us (68045us total) trans 4294973434 rc 0/0 00000100:00100000:2.0:1789760784.884749:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6145 00000100:00100000:2.0:1789760784.884759:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774105327492 time 774148111928 state 0x2 ref 0 00000001:00080000:1.0:1789760784.884833:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973434 is committed 00000100:00100000:2.0:1789760784.891198:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760784.891331:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484353536 00000100:00100000:1.0:1789760784.891346:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484353536 00000100:00100000:1.0:1789760784.891434:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774105327492 time 774148111928 state 0x0 ref 1 00000100:00100000:1.0:1789760784.891454:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774148372344 time 774148111928 state 0x1 ref 1 00000100:00100000:1.0:1789760784.891505:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774148372344 time 774222434886 state 0x0 ref 1 00000100:00100000:1.0:1789760784.891520:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484353536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760784.891599:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25055232, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:1.0:1789760784.891611:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.891631:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:1.0:1789760784.891636:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760784.891643:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25055232->25059327] (req 25055232->25059327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760784.891664:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25055232->25059327] (req 25055232->25059327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760784.943226:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25055232 -> 25059328 00010000:00010000:0.0:1789760784.943253:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25055232->25059327] (req 25055232->25059327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.943269:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25055232->25059327] (req 25055232->25059327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.943279:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25055232->25059327] (req 25055232->25059327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.943287:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760784.943294:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25055232->25059327] (req 25055232->25059327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.943305:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25055232->25059327] (req 25055232->25059327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.943330:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25055232->25059327] (req 25055232->25059327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760784.943339:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a100 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25055232->25059327] (req 25055232->25059327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760784.943467:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484353536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51942us (52278us total) trans 4294973435 rc 0/0 00000100:00100000:0.0:1789760784.943481:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6146 00000100:00100000:0.0:1789760784.943494:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774148372344 time 774222434886 state 0x2 ref 0 00000001:00080000:3.0:1789760784.943585:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973435 is committed 00000100:00100000:0.0:1789760784.948916:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760784.949033:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484353664 00000100:00100000:2.0:1789760784.949046:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484353664 00000100:00100000:2.0:1789760784.949216:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774148372344 time 774222434886 state 0x0 ref 1 00000100:00100000:2.0:1789760784.949235:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774222695302 time 774222434886 state 0x1 ref 1 00000100:00100000:2.0:1789760784.949325:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774222695302 time 774280254405 state 0x0 ref 1 00000100:00100000:2.0:1789760784.949341:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484353664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760784.949416:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25059328, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:2.0:1789760784.949427:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.949447:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:2.0:1789760784.949452:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760784.949460:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25059328->25063423] (req 25059328->25063423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760784.949477:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25059328->25063423] (req 25059328->25063423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760785.007483:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973436 is committed 00002000:00010000:2.0:1789760785.007562:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25059328 -> 25063424 00010000:00010000:2.0:1789760785.007593:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25059328->25063423] (req 25059328->25063423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.007609:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25059328->25063423] (req 25059328->25063423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.007619:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25059328->25063423] (req 25059328->25063423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.007627:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760785.007632:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25059328->25063423] (req 25059328->25063423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.007640:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25059328->25063423] (req 25059328->25063423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.007664:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25059328->25063423] (req 25059328->25063423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.007672:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a107 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25059328->25063423] (req 25059328->25063423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760785.011216:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880ce00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484353664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61858us (62419us total) trans 4294973436 rc 0/0 00000100:00100000:2.0:1789760785.011243:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6147 00000100:00100000:2.0:1789760785.011257:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774222695302 time 774280254405 state 0x2 ref 0 00000100:00100000:2.0:1789760785.014376:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760785.015293:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484353792 00000100:00100000:0.0:1789760785.015310:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484353792 00000100:00100000:0.0:1789760785.015399:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774222695302 time 774280254405 state 0x0 ref 1 00000100:00100000:0.0:1789760785.015419:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774280514821 time 774280254405 state 0x1 ref 1 00000100:00100000:0.0:1789760785.015462:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774280514821 time 774346391986 state 0x0 ref 1 00000100:00100000:0.0:1789760785.015482:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484353792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760785.015563:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25063424, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:0.0:1789760785.015578:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.015599:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:0.0:1789760785.015604:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760785.015611:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25063424->25067519] (req 25063424->25067519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.015628:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25063424->25067519] (req 25063424->25067519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760785.066752:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25063424 -> 25067520 00010000:00010000:2.0:1789760785.066792:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25063424->25067519] (req 25063424->25067519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.066808:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25063424->25067519] (req 25063424->25067519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.066819:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25063424->25067519] (req 25063424->25067519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.066828:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760785.066834:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25063424->25067519] (req 25063424->25067519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.066845:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25063424->25067519] (req 25063424->25067519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.066870:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25063424->25067519] (req 25063424->25067519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.066879:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a10e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25063424->25067519] (req 25063424->25067519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760785.067025:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02d880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484353792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51544us (52660us total) trans 4294973437 rc 0/0 00000100:00100000:2.0:1789760785.067039:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6148 00000100:00100000:2.0:1789760785.067052:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774280514821 time 774346391986 state 0x2 ref 0 00000001:00080000:3.0:1789760785.068332:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973437 is committed 00000100:00100000:3.0:1789760785.069419:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760785.069539:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484353920 00000100:00100000:2.0:1789760785.069554:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484353920 00000100:00100000:2.0:1789760785.069753:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774280514821 time 774346391986 state 0x0 ref 1 00000100:00100000:2.0:1789760785.069784:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774346652402 time 774346391986 state 0x1 ref 1 00000100:00100000:2.0:1789760785.069817:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774346652402 time 774400748011 state 0x0 ref 1 00000100:00100000:2.0:1789760785.069832:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484353920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760785.069895:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25067520, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:2.0:1789760785.069906:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.069926:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:2.0:1789760785.069931:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760785.069939:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25067520->25071615] (req 25067520->25071615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.069969:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25067520->25071615] (req 25067520->25071615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760785.138918:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973438 is committed 00002000:00010000:0.0:1789760785.139255:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25067520 -> 25071616 00010000:00010000:0.0:1789760785.139274:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25067520->25071615] (req 25067520->25071615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.139286:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25067520->25071615] (req 25067520->25071615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.139296:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25067520->25071615] (req 25067520->25071615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.139305:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760785.139310:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25067520->25071615] (req 25067520->25071615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.139320:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25067520->25071615] (req 25067520->25071615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.139346:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25067520->25071615] (req 25067520->25071615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.139354:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a115 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25067520->25071615] (req 25067520->25071615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760785.139767:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484353920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69930us (70349us total) trans 4294973438 rc 0/0 00000100:00100000:0.0:1789760785.140015:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6149 00000100:00100000:0.0:1789760785.140029:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774346652402 time 774400748011 state 0x2 ref 0 00000100:00100000:0.0:1789760785.146951:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760785.147762:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484354048 00000100:00100000:1.0:1789760785.147778:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484354048 00000100:00100000:1.0:1789760785.147871:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774346652402 time 774400748011 state 0x0 ref 1 00000100:00100000:1.0:1789760785.147892:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774401008427 time 774400748011 state 0x1 ref 1 00000100:00100000:1.0:1789760785.147997:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774401008427 time 774478915836 state 0x0 ref 1 00000100:00100000:1.0:1789760785.148024:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484354048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760785.148132:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25071616, mtime: 0x6aad9510, atime: 0x0, ctime: 0x6aad9510, blocks: 0xc007 00010000:00010000:1.0:1789760785.148143:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.148168:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:1.0:1789760785.148173:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760785.148181:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25071616->25075711] (req 25071616->25075711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.148199:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25071616->25075711] (req 25071616->25075711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760785.201114:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973439 is committed 00002000:00010000:1.0:1789760785.201211:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25071616 -> 25075712 00002000:00010000:1.0:1789760785.201216:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760784 -> 1789760785 00002000:00010000:1.0:1789760785.201219:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760784 -> 1789760785 00010000:00010000:1.0:1789760785.201236:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25071616->25075711] (req 25071616->25075711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.201251:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25071616->25075711] (req 25071616->25075711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.201261:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25071616->25075711] (req 25071616->25075711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.201270:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760785.201276:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25071616->25075711] (req 25071616->25075711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.201287:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25071616->25075711] (req 25071616->25075711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.201313:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25071616->25075711] (req 25071616->25075711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.201323:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a11c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25071616->25075711] (req 25071616->25075711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760785.201719:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484354048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53700us (54775us total) trans 4294973439 rc 0/0 00000100:00100000:1.0:1789760785.201732:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6150 00000100:00100000:1.0:1789760785.201745:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774401008427 time 774478915836 state 0x2 ref 0 00000100:00100000:3.0:1789760785.205332:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760785.207418:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484354176 00000100:00100000:0.0:1789760785.207441:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484354176 00000100:00100000:1.0:1789760785.207666:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774401008427 time 774478915836 state 0x0 ref 1 00000100:00100000:1.0:1789760785.207688:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774479176252 time 774478915836 state 0x1 ref 1 00000100:00100000:1.0:1789760785.207725:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774479176252 time 774538655685 state 0x0 ref 1 00000100:00100000:1.0:1789760785.207739:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484354176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760785.207809:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25075712, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:1.0:1789760785.207820:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.207841:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:1.0:1789760785.207847:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760785.207855:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25075712->25079807] (req 25075712->25079807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.207872:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25075712->25079807] (req 25075712->25079807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760785.277935:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973440 is committed 00002000:00010000:0.0:1789760785.278171:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25075712 -> 25079808 00010000:00010000:0.0:1789760785.278187:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25075712->25079807] (req 25075712->25079807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.278201:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25075712->25079807] (req 25075712->25079807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.278210:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25075712->25079807] (req 25075712->25079807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.278218:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760785.278224:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25075712->25079807] (req 25075712->25079807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.278234:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25075712->25079807] (req 25075712->25079807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.278261:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25075712->25079807] (req 25075712->25079807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.278270:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a123 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25075712->25079807] (req 25075712->25079807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760785.278370:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dca80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484354176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 70628us (73046us total) trans 4294973440 rc 0/0 00000100:00100000:0.0:1789760785.278381:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6151 00000100:00100000:0.0:1789760785.278390:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774479176252 time 774538655685 state 0x2 ref 0 00000100:00100000:3.0:1789760785.282166:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760785.283114:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484354304 00000100:00100000:1.0:1789760785.283127:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484354304 00000100:00100000:1.0:1789760785.283195:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774479176252 time 774538655685 state 0x0 ref 1 00000100:00100000:1.0:1789760785.283214:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774538916101 time 774538655685 state 0x1 ref 1 00000100:00100000:1.0:1789760785.283254:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774538916101 time 774614184555 state 0x0 ref 1 00000100:00100000:1.0:1789760785.283268:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484354304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760785.283320:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25079808, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:1.0:1789760785.283329:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.283343:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:1.0:1789760785.283347:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760785.283354:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25079808->25083903] (req 25079808->25083903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.283367:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25079808->25083903] (req 25079808->25083903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.294317:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760785.347130:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973441 is committed 00002000:00010000:3.0:1789760785.347315:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25079808 -> 25083904 00010000:00010000:3.0:1789760785.347330:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25079808->25083903] (req 25079808->25083903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.347342:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25079808->25083903] (req 25079808->25083903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.347351:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25079808->25083903] (req 25079808->25083903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.347358:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760785.347364:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25079808->25083903] (req 25079808->25083903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.347373:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25079808->25083903] (req 25079808->25083903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.347392:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25079808->25083903] (req 25079808->25083903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.347401:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a12a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25079808->25083903] (req 25079808->25083903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760785.347506:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484354304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 64236us (65346us total) trans 4294973441 rc 0/0 00000100:00100000:3.0:1789760785.347517:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6152 00000100:00100000:3.0:1789760785.347528:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774538916101 time 774614184555 state 0x2 ref 0 00000100:00100000:3.0:1789760785.356295:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760785.356405:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484354432 00000100:00100000:2.0:1789760785.356415:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484354432 00000100:00100000:2.0:1789760785.356790:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774538916101 time 774614184555 state 0x0 ref 1 00000100:00100000:2.0:1789760785.356809:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774614444971 time 774614184555 state 0x1 ref 1 00000100:00100000:2.0:1789760785.356846:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774614444971 time 774687776398 state 0x0 ref 1 00000100:00100000:2.0:1789760785.356860:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484354432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760785.356906:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25083904, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:2.0:1789760785.356915:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.356930:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:2.0:1789760785.356935:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760785.356942:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25083904->25087999] (req 25083904->25087999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.356957:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25083904->25087999] (req 25083904->25087999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760785.408855:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973442 is committed 00002000:00010000:3.0:1789760785.408942:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25083904 -> 25088000 00010000:00010000:3.0:1789760785.408958:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25083904->25087999] (req 25083904->25087999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.408971:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25083904->25087999] (req 25083904->25087999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.408980:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25083904->25087999] (req 25083904->25087999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.408998:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760785.409003:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25083904->25087999] (req 25083904->25087999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.409012:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25083904->25087999] (req 25083904->25087999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.409033:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25083904->25087999] (req 25083904->25087999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.409046:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a131 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25083904->25087999] (req 25083904->25087999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760785.409913:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484354432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53047us (53619us total) trans 4294973442 rc 0/0 00000100:00100000:3.0:1789760785.409925:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6153 00000100:00100000:3.0:1789760785.409935:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774614444971 time 774687776398 state 0x2 ref 0 00000100:00100000:1.0:1789760785.418556:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760785.424718:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484354560 00000100:00100000:3.0:1789760785.424735:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484354560 00000100:00100000:3.0:1789760785.424802:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774614444971 time 774687776398 state 0x0 ref 1 00000100:00100000:3.0:1789760785.424819:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774688036814 time 774687776398 state 0x1 ref 1 00000100:00100000:3.0:1789760785.424857:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774688036814 time 774755788338 state 0x0 ref 1 00000100:00100000:3.0:1789760785.424872:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484354560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760785.424924:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25088000, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:3.0:1789760785.424934:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.424950:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:3.0:1789760785.424954:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760785.424961:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25088000->25092095] (req 25088000->25092095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.424981:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25088000->25092095] (req 25088000->25092095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760785.456394:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25088000 -> 25092096 00010000:00010000:1.0:1789760785.456414:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25088000->25092095] (req 25088000->25092095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.456428:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25088000->25092095] (req 25088000->25092095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.456437:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25088000->25092095] (req 25088000->25092095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.456444:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760785.456448:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25088000->25092095] (req 25088000->25092095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.456457:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25088000->25092095] (req 25088000->25092095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.456478:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25088000->25092095] (req 25088000->25092095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.456486:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a138 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25088000->25092095] (req 25088000->25092095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760785.456590:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484354560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31716us (38038us total) trans 4294973443 rc 0/0 00000100:00100000:1.0:1789760785.456600:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6154 00000100:00100000:1.0:1789760785.456611:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774688036814 time 774755788338 state 0x2 ref 0 00000001:00080000:0.0:1789760785.457053:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973443 is committed 00000100:00100000:2.0:1789760785.465389:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760785.465516:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484354688 00000100:00100000:0.0:1789760785.465526:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484354688 00000100:00100000:0.0:1789760785.465585:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774688036814 time 774755788338 state 0x0 ref 1 00000100:00100000:0.0:1789760785.465599:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774756048754 time 774755788338 state 0x1 ref 1 00000100:00100000:0.0:1789760785.465629:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774756048754 time 774796559893 state 0x0 ref 1 00000100:00100000:0.0:1789760785.465643:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484354688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760785.465690:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25092096, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:0.0:1789760785.465699:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.465714:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:0.0:1789760785.465719:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760785.465726:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25092096->25096191] (req 25092096->25096191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.465741:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25092096->25096191] (req 25092096->25096191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760785.518507:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25092096 -> 25096192 00010000:00010000:0.0:1789760785.518529:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25092096->25096191] (req 25092096->25096191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.518540:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25092096->25096191] (req 25092096->25096191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.518548:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25092096->25096191] (req 25092096->25096191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.518555:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760785.518560:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25092096->25096191] (req 25092096->25096191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.518568:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25092096->25096191] (req 25092096->25096191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.518587:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25092096->25096191] (req 25092096->25096191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.518595:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a13f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25092096->25096191] (req 25092096->25096191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760785.518758:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484354688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53112us (53371us total) trans 4294973444 rc 0/0 00000100:00100000:0.0:1789760785.518768:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6155 00000100:00100000:0.0:1789760785.518779:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774756048754 time 774796559893 state 0x2 ref 0 00000001:00080000:0.0:1789760785.519055:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973444 is committed 00000100:00100000:0.0:1789760785.528352:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760785.528775:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484354816 00000100:00100000:3.0:1789760785.528786:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484354816 00000100:00100000:3.0:1789760785.528840:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774756048754 time 774796559893 state 0x0 ref 1 00000100:00100000:3.0:1789760785.528855:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774796820309 time 774796559893 state 0x1 ref 1 00000100:00100000:3.0:1789760785.528921:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774796820309 time 774859850897 state 0x0 ref 1 00000100:00100000:3.0:1789760785.528934:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484354816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760785.528981:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25096192, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:3.0:1789760785.529000:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.529016:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:3.0:1789760785.529020:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760785.529026:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25096192->25100287] (req 25096192->25100287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.529041:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25096192->25100287] (req 25096192->25100287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760785.586458:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973445 is committed 00002000:00010000:3.0:1789760785.586606:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25096192 -> 25100288 00010000:00010000:3.0:1789760785.586622:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25096192->25100287] (req 25096192->25100287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.586636:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25096192->25100287] (req 25096192->25100287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.586644:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25096192->25100287] (req 25096192->25100287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.586652:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760785.586657:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25096192->25100287] (req 25096192->25100287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.586666:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25096192->25100287] (req 25096192->25100287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.586689:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25096192->25100287] (req 25096192->25100287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.586696:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a146 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25096192->25100287] (req 25096192->25100287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760785.586880:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484354816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57944us (58532us total) trans 4294973445 rc 0/0 00000100:00100000:3.0:1789760785.586891:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6156 00000100:00100000:3.0:1789760785.586901:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774796820309 time 774859850897 state 0x2 ref 0 00000100:00100000:3.0:1789760785.593571:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760785.593678:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484354944 00000100:00100000:2.0:1789760785.593688:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484354944 00000100:00100000:2.0:1789760785.593740:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774796820309 time 774859850897 state 0x0 ref 1 00000100:00100000:2.0:1789760785.593754:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774860111313 time 774859850897 state 0x1 ref 1 00000100:00100000:2.0:1789760785.593874:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774860111313 time 774924804219 state 0x0 ref 1 00000100:00100000:2.0:1789760785.593888:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484354944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760785.593935:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25100288, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:2.0:1789760785.593944:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.593960:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:2.0:1789760785.593964:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760785.593970:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25100288->25104383] (req 25100288->25104383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.594154:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25100288->25104383] (req 25100288->25104383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760785.653636:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973446 is committed 00002000:00010000:0.0:1789760785.653651:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25100288 -> 25104384 00010000:00010000:0.0:1789760785.653667:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25100288->25104383] (req 25100288->25104383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.653679:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25100288->25104383] (req 25100288->25104383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.653688:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25100288->25104383] (req 25100288->25104383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.653695:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760785.653700:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25100288->25104383] (req 25100288->25104383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.653709:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25100288->25104383] (req 25100288->25104383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.653728:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25100288->25104383] (req 25100288->25104383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.653736:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a14d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25100288->25104383] (req 25100288->25104383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760785.653831:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484354944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59941us (60266us total) trans 4294973446 rc 0/0 00000100:00100000:0.0:1789760785.653841:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6157 00000100:00100000:0.0:1789760785.653851:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774860111313 time 774924804219 state 0x2 ref 0 00000100:00100000:0.0:1789760785.656825:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760785.656918:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484355072 00000100:00100000:1.0:1789760785.656932:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484355072 00000100:00100000:1.0:1789760785.657013:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774860111313 time 774924804219 state 0x0 ref 1 00000100:00100000:1.0:1789760785.657030:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774925064635 time 774924804219 state 0x1 ref 1 00000100:00100000:1.0:1789760785.657059:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774925064635 time 774987990433 state 0x0 ref 1 00000100:00100000:1.0:1789760785.657097:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760785.657147:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25104384, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:1.0:1789760785.657156:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.657172:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=774s v=(0 0) 00010000:00010000:1.0:1789760785.657177:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760785.657182:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25104384->25108479] (req 25104384->25108479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.657196:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25104384->25108479] (req 25104384->25108479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760785.733694:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973447 is committed 00002000:00010000:1.0:1789760785.733996:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25104384 -> 25108480 00010000:00010000:1.0:1789760785.734011:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25104384->25108479] (req 25104384->25108479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.734024:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25104384->25108479] (req 25104384->25108479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.734033:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25104384->25108479] (req 25104384->25108479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.734040:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760785.734044:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25104384->25108479] (req 25104384->25108479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.734053:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25104384->25108479] (req 25104384->25108479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.734102:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25104384->25108479] (req 25104384->25108479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.734110:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a154 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25104384->25108479] (req 25104384->25108479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760785.734204:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77105us (77392us total) trans 4294973447 rc 0/0 00000100:00100000:1.0:1789760785.734218:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6158 00000100:00100000:1.0:1789760785.734227:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774925064635 time 774987990433 state 0x2 ref 0 00000100:00100000:0.0:1789760785.738276:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760785.738425:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484355200 00000100:00100000:2.0:1789760785.738468:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484355200 00000100:00100000:2.0:1789760785.738522:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774925064635 time 774987990433 state 0x0 ref 1 00000100:00100000:2.0:1789760785.738538:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774988250849 time 774987990433 state 0x1 ref 1 00000100:00100000:2.0:1789760785.738609:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774988250849 time 775069539397 state 0x0 ref 1 00000100:00100000:2.0:1789760785.738626:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760785.738684:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25108480, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:2.0:1789760785.738694:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.738710:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:2.0:1789760785.738715:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760785.738722:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25108480->25112575] (req 25108480->25112575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.738736:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25108480->25112575] (req 25108480->25112575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760785.782016:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973448 is committed 00002000:00010000:0.0:1789760785.782125:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25108480 -> 25112576 00010000:00010000:0.0:1789760785.782144:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25108480->25112575] (req 25108480->25112575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.782157:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25108480->25112575] (req 25108480->25112575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.782166:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25108480->25112575] (req 25108480->25112575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.782173:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760785.782179:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25108480->25112575] (req 25108480->25112575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.782188:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25108480->25112575] (req 25108480->25112575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.782206:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25108480->25112575] (req 25108480->25112575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.782214:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a15b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25108480->25112575] (req 25108480->25112575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760785.782305:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c880d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43677us (44035us total) trans 4294973448 rc 0/0 00000100:00100000:0.0:1789760785.782315:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6159 00000100:00100000:0.0:1789760785.782324:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774988250849 time 775069539397 state 0x2 ref 0 00000100:00100000:2.0:1789760785.787445:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760785.787591:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484355328 00000100:00100000:3.0:1789760785.787602:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484355328 00000100:00100000:3.0:1789760785.787658:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 774988250849 time 775069539397 state 0x0 ref 1 00000100:00100000:3.0:1789760785.787673:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775069799813 time 775069539397 state 0x1 ref 1 00000100:00100000:3.0:1789760785.787698:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775069799813 time 775118629009 state 0x0 ref 1 00000100:00100000:3.0:1789760785.787710:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484355328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760785.787759:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25112576, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:3.0:1789760785.787778:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.787794:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:3.0:1789760785.787798:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760785.787804:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25112576->25116671] (req 25112576->25116671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760785.787817:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25112576->25116671] (req 25112576->25116671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760785.825158:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25112576 -> 25116672 00010000:00010000:0.0:1789760785.825184:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25112576->25116671] (req 25112576->25116671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.825199:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25112576->25116671] (req 25112576->25116671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.825214:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25112576->25116671] (req 25112576->25116671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.825223:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760785.825229:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25112576->25116671] (req 25112576->25116671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.825239:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25112576->25116671] (req 25112576->25116671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.825266:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25112576->25116671] (req 25112576->25116671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.825274:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a162 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25112576->25116671] (req 25112576->25116671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760785.825399:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484355328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37683us (37956us total) trans 4294973449 rc 0/0 00000100:00100000:0.0:1789760785.825411:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6160 00000100:00100000:0.0:1789760785.825422:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775069799813 time 775118629009 state 0x2 ref 0 00000001:00080000:1.0:1789760785.825762:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973449 is committed 00000100:00100000:1.0:1789760785.831758:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760785.831847:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484355456 00000100:00100000:0.0:1789760785.831860:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484355456 00000100:00100000:0.0:1789760785.831941:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775069799813 time 775118629009 state 0x0 ref 1 00000100:00100000:0.0:1789760785.831970:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775118889425 time 775118629009 state 0x1 ref 1 00000100:00100000:0.0:1789760785.832020:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775118889425 time 775162950614 state 0x0 ref 1 00000100:00100000:0.0:1789760785.832035:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760785.832144:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25116672, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:0.0:1789760785.832155:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.832173:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:0.0:1789760785.832178:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760785.832185:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25116672->25120767] (req 25116672->25120767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760785.832202:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25116672->25120767] (req 25116672->25120767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760785.874494:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973450 is committed 00002000:00010000:2.0:1789760785.874506:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25116672 -> 25120768 00010000:00010000:2.0:1789760785.874527:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25116672->25120767] (req 25116672->25120767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.874602:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25116672->25120767] (req 25116672->25120767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.874614:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25116672->25120767] (req 25116672->25120767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.874622:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760785.874626:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25116672->25120767] (req 25116672->25120767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.874634:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25116672->25120767] (req 25116672->25120767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.874656:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25116672->25120767] (req 25116672->25120767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.874661:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c78fc8fe/0x2cf382639620a169 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25116672->25120767] (req 25116672->25120767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760785.874769:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42731us (43018us total) trans 4294973450 rc 0/0 00000100:00100000:2.0:1789760785.874782:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6161 00000100:00100000:2.0:1789760785.874793:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775118889425 time 775162950614 state 0x2 ref 0 00000100:00100000:2.0:1789760785.884655:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760785.884802:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484355584 00000100:00100000:1.0:1789760785.884816:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484355584 00000100:00100000:1.0:1789760785.884884:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775118889425 time 775162950614 state 0x0 ref 1 00000100:00100000:1.0:1789760785.884903:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775163211030 time 775162950614 state 0x1 ref 1 00000100:00100000:1.0:1789760785.884936:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775163211030 time 775215866582 state 0x0 ref 1 00000100:00100000:1.0:1789760785.885091:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760785.885162:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25120768, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:1.0:1789760785.885173:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.885194:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:1.0:1789760785.885199:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760785.885206:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25120768->25124863] (req 25120768->25124863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760785.885223:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25120768->25124863] (req 25120768->25124863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760785.947539:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973451 is committed 00002000:00010000:2.0:1789760785.947722:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25120768 -> 25124864 00010000:00010000:2.0:1789760785.947743:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25120768->25124863] (req 25120768->25124863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.947758:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25120768->25124863] (req 25120768->25124863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.947828:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25120768->25124863] (req 25120768->25124863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.947844:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760785.947850:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25120768->25124863] (req 25120768->25124863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.947860:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25120768->25124863] (req 25120768->25124863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.947886:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25120768->25124863] (req 25120768->25124863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.947893:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a170 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25120768->25124863] (req 25120768->25124863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760785.948308:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df02c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63109us (63416us total) trans 4294973451 rc 0/0 00000100:00100000:2.0:1789760785.948324:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6162 00000100:00100000:2.0:1789760785.948337:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775163211030 time 775215866582 state 0x2 ref 0 00000100:00100000:3.0:1789760785.953185:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760785.953270:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484355712 00000100:00100000:2.0:1789760785.953283:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484355712 00000100:00100000:2.0:1789760785.953356:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775163211030 time 775215866582 state 0x0 ref 1 00000100:00100000:2.0:1789760785.953383:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775216126998 time 775215866582 state 0x1 ref 1 00000100:00100000:2.0:1789760785.953416:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775216126998 time 775284346943 state 0x0 ref 1 00000100:00100000:2.0:1789760785.953435:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760785.953500:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25124864, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:2.0:1789760785.953510:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.953530:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:2.0:1789760785.953535:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760785.953544:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25124864->25128959] (req 25124864->25128959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760785.953560:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25124864->25128959] (req 25124864->25128959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760786.011478:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25124864 -> 25128960 00010000:00010000:2.0:1789760786.011508:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25124864->25128959] (req 25124864->25128959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.011525:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25124864->25128959] (req 25124864->25128959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.011535:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25124864->25128959] (req 25124864->25128959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.011543:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760786.011549:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25124864->25128959] (req 25124864->25128959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.011560:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25124864->25128959] (req 25124864->25128959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.011584:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25124864->25128959] (req 25124864->25128959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.011592:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a177 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25124864->25128959] (req 25124864->25128959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760786.011719:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484355712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58280us (58542us total) trans 4294973452 rc 0/0 00000100:00100000:2.0:1789760786.011732:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6163 00000100:00100000:2.0:1789760786.011743:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775216126998 time 775284346943 state 0x2 ref 0 00000001:00080000:3.0:1789760786.012658:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973452 is committed 00000100:00100000:1.0:1789760786.022947:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760786.023438:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484355840 00000100:00100000:0.0:1789760786.023453:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484355840 00000100:00100000:1.0:1789760786.023672:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775216126998 time 775284346943 state 0x0 ref 1 00000100:00100000:1.0:1789760786.023691:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775284607359 time 775284346943 state 0x1 ref 1 00000100:00100000:1.0:1789760786.023786:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775284607359 time 775354715816 state 0x0 ref 1 00000100:00100000:1.0:1789760786.023803:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760786.023872:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25128960, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:1.0:1789760786.023883:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.023902:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:1.0:1789760786.023907:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760786.023914:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25128960->25133055] (req 25128960->25133055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.023931:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25128960->25133055] (req 25128960->25133055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760786.024716:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:3.0:1789760786.070662:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25128960 -> 25133056 00010000:00010000:3.0:1789760786.070691:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25128960->25133055] (req 25128960->25133055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.070707:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25128960->25133055] (req 25128960->25133055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.070717:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25128960->25133055] (req 25128960->25133055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.070725:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760786.070731:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25128960->25133055] (req 25128960->25133055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.070742:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25128960->25133055] (req 25128960->25133055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.070836:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25128960->25133055] (req 25128960->25133055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.070845:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a17e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25128960->25133055] (req 25128960->25133055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760786.070993:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e84a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484355840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47171us (48038us total) trans 4294973453 rc 0/0 00000100:00100000:3.0:1789760786.071163:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6164 00000100:00100000:3.0:1789760786.071179:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775284607359 time 775354715816 state 0x2 ref 0 00000001:00080000:3.0:1789760786.071238:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973453 is committed 00000100:00100000:1.0:1789760786.074176:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760786.074378:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484355968 00000100:00100000:0.0:1789760786.074392:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484355968 00000100:00100000:0.0:1789760786.074478:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775284607359 time 775354715816 state 0x0 ref 1 00000100:00100000:0.0:1789760786.074496:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775354976232 time 775354715816 state 0x1 ref 1 00000100:00100000:0.0:1789760786.074534:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775354976232 time 775405464616 state 0x0 ref 1 00000100:00100000:0.0:1789760786.074550:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e87800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760786.074618:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25133056, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:0.0:1789760786.074629:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.074649:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:0.0:1789760786.074654:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760786.074661:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25133056->25137151] (req 25133056->25137151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.074684:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25133056->25137151] (req 25133056->25137151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760786.115930:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973454 is committed 00002000:00010000:0.0:1789760786.116153:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25133056 -> 25137152 00010000:00010000:0.0:1789760786.116177:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25133056->25137151] (req 25133056->25137151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.116193:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25133056->25137151] (req 25133056->25137151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.116205:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25133056->25137151] (req 25133056->25137151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.116213:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760786.116218:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25133056->25137151] (req 25133056->25137151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.116227:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25133056->25137151] (req 25133056->25137151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.116253:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25133056->25137151] (req 25133056->25137151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.116261:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a185 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25133056->25137151] (req 25133056->25137151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760786.116425:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e87800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484355968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41871us (42257us total) trans 4294973454 rc 0/0 00000100:00100000:0.0:1789760786.116436:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6165 00000100:00100000:0.0:1789760786.116449:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775354976232 time 775405464616 state 0x2 ref 0 00000100:00100000:2.0:1789760786.127171:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760786.127327:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484356096 00000100:00100000:3.0:1789760786.127342:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484356096 00000100:00100000:3.0:1789760786.127416:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775354976232 time 775405464616 state 0x0 ref 1 00000100:00100000:3.0:1789760786.127436:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775405725032 time 775405464616 state 0x1 ref 1 00000100:00100000:3.0:1789760786.127521:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775405725032 time 775458450981 state 0x0 ref 1 00000100:00100000:3.0:1789760786.127546:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484356096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760786.127614:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25137152, mtime: 0x6aad9511, atime: 0x0, ctime: 0x6aad9511, blocks: 0xc007 00010000:00010000:3.0:1789760786.127625:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.127645:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:3.0:1789760786.127650:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760786.127658:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25137152->25141247] (req 25137152->25141247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.127674:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25137152->25141247] (req 25137152->25141247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760786.160935:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973455 is committed 00002000:00010000:0.0:1789760786.161252:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25137152 -> 25141248 00002000:00010000:0.0:1789760786.161258:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760785 -> 1789760786 00002000:00010000:0.0:1789760786.161260:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760785 -> 1789760786 00010000:00010000:0.0:1789760786.161279:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25137152->25141247] (req 25137152->25141247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.161293:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25137152->25141247] (req 25137152->25141247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.161303:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25137152->25141247] (req 25137152->25141247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.161312:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760786.161317:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25137152->25141247] (req 25137152->25141247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.161328:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25137152->25141247] (req 25137152->25141247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.161353:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25137152->25141247] (req 25137152->25141247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.161362:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a18c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25137152->25141247] (req 25137152->25141247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760786.161518:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484356096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33977us (34354us total) trans 4294973455 rc 0/0 00000100:00100000:0.0:1789760786.161531:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6166 00000100:00100000:0.0:1789760786.161543:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775405725032 time 775458450981 state 0x2 ref 0 00000100:00100000:2.0:1789760786.166313:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760786.166647:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484356224 00000100:00100000:3.0:1789760786.166662:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484356224 00000100:00100000:3.0:1789760786.166741:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775405725032 time 775458450981 state 0x0 ref 1 00000100:00100000:3.0:1789760786.166774:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775458711397 time 775458450981 state 0x1 ref 1 00000100:00100000:3.0:1789760786.166805:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775458711397 time 775497735955 state 0x0 ref 1 00000100:00100000:3.0:1789760786.166820:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484356224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760786.166887:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25141248, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc007 00010000:00010000:3.0:1789760786.166909:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.166931:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:3.0:1789760786.166935:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760786.166943:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25141248->25145343] (req 25141248->25145343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.166961:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25141248->25145343] (req 25141248->25145343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760786.211588:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973456 is committed 00002000:00010000:2.0:1789760786.211837:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25141248 -> 25145344 00010000:00010000:2.0:1789760786.211859:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25141248->25145343] (req 25141248->25145343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.211875:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25141248->25145343] (req 25141248->25145343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.211886:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25141248->25145343] (req 25141248->25145343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.211895:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760786.211901:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25141248->25145343] (req 25141248->25145343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.211911:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25141248->25145343] (req 25141248->25145343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.212156:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25141248->25145343] (req 25141248->25145343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.212164:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a193 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25141248->25145343] (req 25141248->25145343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760786.212325:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484356224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45500us (46029us total) trans 4294973456 rc 0/0 00000100:00100000:2.0:1789760786.212337:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6167 00000100:00100000:2.0:1789760786.212350:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775458711397 time 775497735955 state 0x2 ref 0 00000100:00100000:1.0:1789760786.217742:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760786.218167:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484356352 00000100:00100000:3.0:1789760786.218182:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484356352 00000100:00100000:3.0:1789760786.218262:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775458711397 time 775497735955 state 0x0 ref 1 00000100:00100000:3.0:1789760786.218282:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775497996371 time 775497735955 state 0x1 ref 1 00000100:00100000:3.0:1789760786.218315:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775497996371 time 775549246344 state 0x0 ref 1 00000100:00100000:3.0:1789760786.218329:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484356352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760786.218393:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25145344, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc007 00010000:00010000:3.0:1789760786.218403:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.218422:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:3.0:1789760786.218427:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760786.218435:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25145344->25149439] (req 25145344->25149439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.218451:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25145344->25149439] (req 25145344->25149439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760786.263517:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25145344 -> 25149440 00010000:00010000:2.0:1789760786.263546:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25145344->25149439] (req 25145344->25149439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.263562:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25145344->25149439] (req 25145344->25149439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.263579:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25145344->25149439] (req 25145344->25149439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.263587:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760786.263593:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25145344->25149439] (req 25145344->25149439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.263604:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25145344->25149439] (req 25145344->25149439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.263629:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25145344->25149439] (req 25145344->25149439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.263638:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a19a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25145344->25149439] (req 25145344->25149439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760786.263964:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c3e86680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484356352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45447us (46047us total) trans 4294973457 rc 0/0 00000100:00100000:2.0:1789760786.263978:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6168 00000100:00100000:2.0:1789760786.263998:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775497996371 time 775549246344 state 0x2 ref 0 00000001:00080000:1.0:1789760786.264649:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973457 is committed 00000100:00100000:3.0:1789760786.267023:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760786.267387:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484356480 00000100:00100000:0.0:1789760786.267399:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484356480 00000100:00100000:0.0:1789760786.267473:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775497996371 time 775549246344 state 0x0 ref 1 00000100:00100000:0.0:1789760786.267489:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775549506760 time 775549246344 state 0x1 ref 1 00000100:00100000:0.0:1789760786.267517:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775549506760 time 775598447922 state 0x0 ref 1 00000100:00100000:0.0:1789760786.267533:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484356480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760786.267598:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25149440, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc007 00010000:00010000:0.0:1789760786.267610:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.267630:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:0.0:1789760786.267636:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760786.267644:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25149440->25153535] (req 25149440->25153535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.267661:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25149440->25153535] (req 25149440->25153535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.317314:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760786.347272:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973458 is committed 00002000:00010000:0.0:1789760786.347469:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25149440 -> 25153536 00010000:00010000:0.0:1789760786.347490:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25149440->25153535] (req 25149440->25153535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.347506:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25149440->25153535] (req 25149440->25153535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.347516:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25149440->25153535] (req 25149440->25153535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.347525:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760786.347530:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25149440->25153535] (req 25149440->25153535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.347541:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25149440->25153535] (req 25149440->25153535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.347567:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25149440->25153535] (req 25149440->25153535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.347577:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a1a1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25149440->25153535] (req 25149440->25153535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760786.347703:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484356480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 80168us (80688us total) trans 4294973458 rc 0/0 00000100:00100000:0.0:1789760786.347716:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6169 00000100:00100000:0.0:1789760786.347729:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775549506760 time 775598447922 state 0x2 ref 0 00000100:00100000:0.0:1789760786.351329:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760786.351452:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484356608 00000100:00100000:3.0:1789760786.351465:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484356608 00000100:00100000:3.0:1789760786.351535:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775549506760 time 775598447922 state 0x0 ref 1 00000100:00100000:3.0:1789760786.351554:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775598708338 time 775598447922 state 0x1 ref 1 00000100:00100000:3.0:1789760786.351590:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775598708338 time 775682520564 state 0x0 ref 1 00000100:00100000:3.0:1789760786.351605:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484356608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760786.351674:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25153536, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc007 00010000:00010000:3.0:1789760786.351684:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.351711:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:3.0:1789760786.351716:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760786.351735:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25153536->25157631] (req 25153536->25157631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.351753:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25153536->25157631] (req 25153536->25157631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760786.383635:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25153536 -> 25157632 00010000:00010000:3.0:1789760786.383663:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25153536->25157631] (req 25153536->25157631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.383680:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25153536->25157631] (req 25153536->25157631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.383690:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25153536->25157631] (req 25153536->25157631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.383698:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760786.383704:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25153536->25157631] (req 25153536->25157631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.383714:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25153536->25157631] (req 25153536->25157631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.383739:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25153536->25157631] (req 25153536->25157631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.383750:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a1a8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25153536->25157631] (req 25153536->25157631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760786.383994:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973459 is committed 00000100:00100000:3.0:1789760786.384051:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c11c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484356608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32441us (32732us total) trans 4294973459 rc 0/0 00000100:00100000:3.0:1789760786.384063:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6170 00000100:00100000:3.0:1789760786.384119:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775598708338 time 775682520564 state 0x2 ref 0 00000100:00100000:1.0:1789760786.391536:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760786.391649:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484356736 00000100:00100000:2.0:1789760786.391663:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484356736 00000100:00100000:2.0:1789760786.391737:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775598708338 time 775682520564 state 0x0 ref 1 00000100:00100000:2.0:1789760786.391762:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775682780980 time 775682520564 state 0x1 ref 1 00000100:00100000:2.0:1789760786.391806:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775682780980 time 775722736257 state 0x0 ref 1 00000100:00100000:2.0:1789760786.391820:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484356736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760786.391888:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25157632, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc007 00010000:00010000:2.0:1789760786.391899:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.391920:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:2.0:1789760786.391925:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760786.391933:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25157632->25161727] (req 25157632->25161727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.391948:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25157632->25161727] (req 25157632->25161727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760786.435244:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973460 is committed 00002000:00010000:0.0:1789760786.435421:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25157632 -> 25161728 00010000:00010000:0.0:1789760786.435440:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25157632->25161727] (req 25157632->25161727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.435456:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25157632->25161727] (req 25157632->25161727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.435467:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25157632->25161727] (req 25157632->25161727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.435475:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760786.435481:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25157632->25161727] (req 25157632->25161727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.435491:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25157632->25161727] (req 25157632->25161727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.435521:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25157632->25161727] (req 25157632->25161727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.435530:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a1af lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25157632->25161727] (req 25157632->25161727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760786.435834:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484356736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44008us (44303us total) trans 4294973460 rc 0/0 00000100:00100000:0.0:1789760786.435848:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6171 00000100:00100000:0.0:1789760786.435861:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775682780980 time 775722736257 state 0x2 ref 0 00000100:00100000:0.0:1789760786.441251:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760786.441362:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484356864 00000100:00100000:2.0:1789760786.441376:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484356864 00000100:00100000:2.0:1789760786.441459:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775682780980 time 775722736257 state 0x0 ref 1 00000100:00100000:2.0:1789760786.441477:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775722996673 time 775722736257 state 0x1 ref 1 00000100:00100000:2.0:1789760786.441513:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775722996673 time 775772443562 state 0x0 ref 1 00000100:00100000:2.0:1789760786.441527:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b82d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484356864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760786.441609:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25161728, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc007 00010000:00010000:2.0:1789760786.441621:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.441640:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:2.0:1789760786.441646:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760786.441654:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25161728->25165823] (req 25161728->25165823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.441671:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25161728->25165823] (req 25161728->25165823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760786.475961:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25161728 -> 25165824 00010000:00010000:2.0:1789760786.475998:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25161728->25165823] (req 25161728->25165823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.476014:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25161728->25165823] (req 25161728->25165823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.476025:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25161728->25165823] (req 25161728->25165823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.476035:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760786.476041:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25161728->25165823] (req 25161728->25165823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.476051:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25161728->25165823] (req 25161728->25165823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.476118:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25161728->25165823] (req 25161728->25165823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.476129:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1b6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25161728->25165823] (req 25161728->25165823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760786.476255:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b82d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484356864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34723us (35012us total) trans 4294973461 rc 0/0 00000100:00100000:2.0:1789760786.476270:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6172 00000100:00100000:2.0:1789760786.476282:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775722996673 time 775772443562 state 0x2 ref 0 00000001:00080000:0.0:1789760786.476701:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973461 is committed 00000100:00100000:1.0:1789760786.483612:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760786.485268:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484356992 00000100:00100000:0.0:1789760786.485286:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484356992 00000100:00100000:1.0:1789760786.485571:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775722996673 time 775772443562 state 0x0 ref 1 00000100:00100000:1.0:1789760786.485592:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775772703978 time 775772443562 state 0x1 ref 1 00000100:00100000:1.0:1789760786.485667:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775772703978 time 775816597345 state 0x0 ref 1 00000100:00100000:1.0:1789760786.485684:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484356992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760786.485749:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25165824, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc007 00010000:00010000:1.0:1789760786.485759:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.485778:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:1.0:1789760786.485783:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760786.485791:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25165824->25169919] (req 25165824->25169919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.485807:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25165824->25169919] (req 25165824->25169919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760786.533929:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973462 is committed 00002000:00010000:0.0:1789760786.534230:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25165824 -> 25169920 00002000:00010000:0.0:1789760786.534236:0:8106:0:(ofd_lvb.c:331:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb blocks from disk: 49159 -> 51207 00010000:00010000:0.0:1789760786.534253:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25165824->25169919] (req 25165824->25169919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.534268:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25165824->25169919] (req 25165824->25169919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.534278:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25165824->25169919] (req 25165824->25169919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.534291:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760786.534296:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25165824->25169919] (req 25165824->25169919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.534305:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25165824->25169919] (req 25165824->25169919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.534332:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25165824->25169919] (req 25165824->25169919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.534340:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a1bd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25165824->25169919] (req 25165824->25169919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760786.534464:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484356992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48777us (50863us total) trans 4294973462 rc 0/0 00000100:00100000:0.0:1789760786.534476:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6173 00000100:00100000:0.0:1789760786.534488:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775772703978 time 775816597345 state 0x2 ref 0 00000100:00100000:1.0:1789760786.538230:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760786.538376:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484357120 00000100:00100000:3.0:1789760786.538389:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484357120 00000100:00100000:3.0:1789760786.538471:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775772703978 time 775816597345 state 0x0 ref 1 00000100:00100000:3.0:1789760786.538491:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775816857761 time 775816597345 state 0x1 ref 1 00000100:00100000:3.0:1789760786.538616:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775816857761 time 775869545101 state 0x0 ref 1 00000100:00100000:3.0:1789760786.538641:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484357120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760786.538704:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25169920, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:3.0:1789760786.538716:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.538737:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:3.0:1789760786.538743:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760786.538751:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25169920->25174015] (req 25169920->25174015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.538782:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25169920->25174015] (req 25169920->25174015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760786.589152:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25169920 -> 25174016 00010000:00010000:1.0:1789760786.589183:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25169920->25174015] (req 25169920->25174015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.589206:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25169920->25174015] (req 25169920->25174015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.589217:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25169920->25174015] (req 25169920->25174015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.589225:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760786.589230:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25169920->25174015] (req 25169920->25174015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.589241:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25169920->25174015] (req 25169920->25174015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.589267:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25169920->25174015] (req 25169920->25174015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.589275:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a1c4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25169920->25174015] (req 25169920->25174015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760786.589386:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484357120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50742us (51166us total) trans 4294973463 rc 0/0 00000100:00100000:1.0:1789760786.589399:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6174 00000100:00100000:1.0:1789760786.589413:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775816857761 time 775869545101 state 0x2 ref 0 00000001:00080000:2.0:1789760786.590117:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973463 is committed 00000100:00100000:3.0:1789760786.595591:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760786.595889:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484357248 00000100:00100000:0.0:1789760786.595903:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484357248 00000100:00100000:0.0:1789760786.595996:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775816857761 time 775869545101 state 0x0 ref 1 00000100:00100000:0.0:1789760786.596015:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775869805517 time 775869545101 state 0x1 ref 1 00000100:00100000:0.0:1789760786.596233:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775869805517 time 775927161896 state 0x0 ref 1 00000100:00100000:0.0:1789760786.596254:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484357248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760786.596326:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25174016, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:0.0:1789760786.596337:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.596357:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:0.0:1789760786.596362:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760786.596369:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25174016->25178111] (req 25174016->25178111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.596387:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25174016->25178111] (req 25174016->25178111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760786.634243:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973464 is committed 00002000:00010000:0.0:1789760786.634366:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25174016 -> 25178112 00010000:00010000:0.0:1789760786.634640:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25174016->25178111] (req 25174016->25178111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.634658:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25174016->25178111] (req 25174016->25178111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.634669:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25174016->25178111] (req 25174016->25178111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.634676:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760786.634682:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25174016->25178111] (req 25174016->25178111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.634692:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25174016->25178111] (req 25174016->25178111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.634718:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25174016->25178111] (req 25174016->25178111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.634727:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a1cb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25174016->25178111] (req 25174016->25178111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760786.634899:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484357248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38643us (39386us total) trans 4294973464 rc 0/0 00000100:00100000:0.0:1789760786.634911:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6175 00000100:00100000:0.0:1789760786.634922:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775869805517 time 775927161896 state 0x2 ref 0 00000100:00100000:0.0:1789760786.643377:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760786.643500:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484357376 00000100:00100000:2.0:1789760786.643515:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484357376 00000100:00100000:2.0:1789760786.643606:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775869805517 time 775927161896 state 0x0 ref 1 00000100:00100000:2.0:1789760786.643627:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775927422312 time 775927161896 state 0x1 ref 1 00000100:00100000:2.0:1789760786.643667:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775927422312 time 775974597461 state 0x0 ref 1 00000100:00100000:2.0:1789760786.643682:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484357376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760786.643750:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25178112, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:2.0:1789760786.643767:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.643792:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=775s v=(0 0) 00010000:00010000:2.0:1789760786.643797:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760786.643805:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25178112->25182207] (req 25178112->25182207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.643823:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25178112->25182207] (req 25178112->25182207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760786.695390:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25178112 -> 25182208 00010000:00010000:1.0:1789760786.695417:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25178112->25182207] (req 25178112->25182207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.695432:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25178112->25182207] (req 25178112->25182207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.695452:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25178112->25182207] (req 25178112->25182207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.695461:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760786.695467:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25178112->25182207] (req 25178112->25182207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.695478:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25178112->25182207] (req 25178112->25182207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.695505:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25178112->25182207] (req 25178112->25182207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.695513:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a1d2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25178112->25182207] (req 25178112->25182207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760786.695635:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b83100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484357376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51948us (52273us total) trans 4294973465 rc 0/0 00000100:00100000:1.0:1789760786.695648:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6176 00000100:00100000:1.0:1789760786.695660:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775927422312 time 775974597461 state 0x2 ref 0 00000001:00080000:2.0:1789760786.697203:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973465 is committed 00000100:00100000:1.0:1789760786.705339:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760786.705437:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484357504 00000100:00100000:0.0:1789760786.705457:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484357504 00000100:00100000:0.0:1789760786.705538:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775927422312 time 775974597461 state 0x0 ref 1 00000100:00100000:0.0:1789760786.705558:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775974857877 time 775974597461 state 0x1 ref 1 00000100:00100000:0.0:1789760786.705594:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775974857877 time 776036525184 state 0x0 ref 1 00000100:00100000:0.0:1789760786.705609:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484357504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760786.705682:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25182208, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:0.0:1789760786.705695:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.705716:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:0.0:1789760786.705722:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760786.705729:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25182208->25186303] (req 25182208->25186303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.705747:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25182208->25186303] (req 25182208->25186303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760786.762726:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973466 is committed 00002000:00010000:0.0:1789760786.762892:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25182208 -> 25186304 00010000:00010000:0.0:1789760786.762913:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25182208->25186303] (req 25182208->25186303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.762929:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25182208->25186303] (req 25182208->25186303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.762939:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25182208->25186303] (req 25182208->25186303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.762946:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760786.762952:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25182208->25186303] (req 25182208->25186303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.762963:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25182208->25186303] (req 25182208->25186303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.763003:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25182208->25186303] (req 25182208->25186303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.763012:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a1d9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25182208->25186303] (req 25182208->25186303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760786.763201:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484357504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57588us (57869us total) trans 4294973466 rc 0/0 00000100:00100000:0.0:1789760786.763213:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6177 00000100:00100000:0.0:1789760786.763225:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775974857877 time 776036525184 state 0x2 ref 0 00000100:00100000:0.0:1789760786.766139:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760786.767049:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484357632 00000100:00100000:1.0:1789760786.767090:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484357632 00000100:00100000:1.0:1789760786.767172:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 775974857877 time 776036525184 state 0x0 ref 1 00000100:00100000:1.0:1789760786.767192:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776036785600 time 776036525184 state 0x1 ref 1 00000100:00100000:1.0:1789760786.767233:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776036785600 time 776098162873 state 0x0 ref 1 00000100:00100000:1.0:1789760786.767248:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b80a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484357632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760786.767316:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25186304, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:1.0:1789760786.767327:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.767347:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:1.0:1789760786.767352:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760786.767361:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25186304->25190399] (req 25186304->25190399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760786.767377:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25186304->25190399] (req 25186304->25190399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760786.844093:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973467 is committed 00002000:00010000:2.0:1789760786.844581:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25186304 -> 25190400 00010000:00010000:2.0:1789760786.844600:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25186304->25190399] (req 25186304->25190399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.844614:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25186304->25190399] (req 25186304->25190399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.844624:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25186304->25190399] (req 25186304->25190399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.844632:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760786.844638:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25186304->25190399] (req 25186304->25190399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.844649:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25186304->25190399] (req 25186304->25190399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.844683:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25186304->25190399] (req 25186304->25190399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.844692:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a1e0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25186304->25190399] (req 25186304->25190399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760786.844816:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b80a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484357632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77564us (78682us total) trans 4294973467 rc 0/0 00000100:00100000:2.0:1789760786.844829:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6178 00000100:00100000:2.0:1789760786.844841:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776036785600 time 776098162873 state 0x2 ref 0 00000100:00100000:0.0:1789760786.857287:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760786.857402:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484357760 00000100:00100000:2.0:1789760786.857415:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484357760 00000100:00100000:2.0:1789760786.857487:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776036785600 time 776098162873 state 0x0 ref 1 00000100:00100000:2.0:1789760786.857506:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776098423289 time 776098162873 state 0x1 ref 1 00000100:00100000:2.0:1789760786.857542:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776098423289 time 776188473048 state 0x0 ref 1 00000100:00100000:2.0:1789760786.857557:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484357760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760786.857622:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25190400, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:2.0:1789760786.857633:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.857653:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:2.0:1789760786.857660:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760786.857667:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25190400->25194495] (req 25190400->25194495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.857684:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25190400->25194495] (req 25190400->25194495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760786.905420:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25190400 -> 25194496 00010000:00010000:2.0:1789760786.905450:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25190400->25194495] (req 25190400->25194495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.905467:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25190400->25194495] (req 25190400->25194495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.905477:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25190400->25194495] (req 25190400->25194495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.905485:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760786.905492:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25190400->25194495] (req 25190400->25194495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.905502:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25190400->25194495] (req 25190400->25194495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.905530:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25190400->25194495] (req 25190400->25194495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.905539:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1e7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25190400->25194495] (req 25190400->25194495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760786.905668:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484357760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48106us (48387us total) trans 4294973468 rc 0/0 00000100:00100000:2.0:1789760786.905680:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6179 00000100:00100000:2.0:1789760786.905693:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776098423289 time 776188473048 state 0x2 ref 0 00000001:00080000:3.0:1789760786.905766:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973468 is committed 00000100:00100000:0.0:1789760786.917726:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760786.917850:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484357888 00000100:00100000:2.0:1789760786.917866:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484357888 00000100:00100000:2.0:1789760786.917948:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776098423289 time 776188473048 state 0x0 ref 1 00000100:00100000:2.0:1789760786.917968:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776188733464 time 776188473048 state 0x1 ref 1 00000100:00100000:2.0:1789760786.918022:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776188733464 time 776248952183 state 0x0 ref 1 00000100:00100000:2.0:1789760786.918037:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484357888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760786.918139:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25194496, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:2.0:1789760786.918151:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.918172:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:2.0:1789760786.918177:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760786.918186:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25194496->25198591] (req 25194496->25198591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760786.918206:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25194496->25198591] (req 25194496->25198591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760786.970939:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973469 is committed 00002000:00010000:3.0:1789760786.971362:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25194496 -> 25198592 00010000:00010000:3.0:1789760786.971393:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25194496->25198591] (req 25194496->25198591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.971409:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25194496->25198591] (req 25194496->25198591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.971420:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25194496->25198591] (req 25194496->25198591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.971429:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760786.971434:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25194496->25198591] (req 25194496->25198591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.971444:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25194496->25198591] (req 25194496->25198591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.971470:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25194496->25198591] (req 25194496->25198591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760786.971483:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a1ee lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25194496->25198591] (req 25194496->25198591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760786.971609:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484357888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53568us (53891us total) trans 4294973469 rc 0/0 00000100:00100000:3.0:1789760786.971623:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6180 00000100:00100000:3.0:1789760786.971636:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776188733464 time 776248952183 state 0x2 ref 0 00000100:00100000:0.0:1789760786.975449:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760786.975868:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484358016 00000100:00100000:2.0:1789760786.975883:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484358016 00000100:00100000:2.0:1789760786.976266:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776188733464 time 776248952183 state 0x0 ref 1 00000100:00100000:2.0:1789760786.976287:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776249212599 time 776248952183 state 0x1 ref 1 00000100:00100000:0.0:1789760786.976390:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776249212599 time 776307319082 state 0x0 ref 1 00000100:00100000:0.0:1789760786.976416:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706a450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484358016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760786.976482:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25198592, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:0.0:1789760786.976493:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.976513:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:0.0:1789760786.976517:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760786.976525:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25198592->25202687] (req 25198592->25202687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760786.976541:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25198592->25202687] (req 25198592->25202687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760787.045484:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973470 is committed 00002000:00010000:1.0:1789760787.045594:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25198592 -> 25202688 00010000:00010000:1.0:1789760787.045615:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25198592->25202687] (req 25198592->25202687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.045636:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25198592->25202687] (req 25198592->25202687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.045648:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25198592->25202687] (req 25198592->25202687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.045656:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760787.045662:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25198592->25202687] (req 25198592->25202687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.045672:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25198592->25202687] (req 25198592->25202687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.045697:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25198592->25202687] (req 25198592->25202687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.045705:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a1f5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25198592->25202687] (req 25198592->25202687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760787.045935:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706a450 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484358016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69524us (70497us total) trans 4294973470 rc 0/0 00000100:00100000:1.0:1789760787.045948:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6181 00000100:00100000:1.0:1789760787.045960:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776249212599 time 776307319082 state 0x2 ref 0 00000100:00100000:0.0:1789760787.053230:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760787.053326:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484358144 00000100:00100000:1.0:1789760787.053340:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484358144 00000100:00100000:1.0:1789760787.053411:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776249212599 time 776307319082 state 0x0 ref 1 00000100:00100000:1.0:1789760787.053548:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776307579498 time 776307319082 state 0x1 ref 1 00000100:00100000:1.0:1789760787.053628:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776307579498 time 776384558218 state 0x0 ref 1 00000100:00100000:1.0:1789760787.053645:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484358144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760787.053710:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25202688, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:1.0:1789760787.053721:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.053740:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:1.0:1789760787.053745:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760787.053753:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25202688->25206783] (req 25202688->25206783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.053781:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25202688->25206783] (req 25202688->25206783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760787.110502:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973471 is committed 00002000:00010000:0.0:1789760787.110626:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25202688 -> 25206784 00010000:00010000:0.0:1789760787.110646:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25202688->25206783] (req 25202688->25206783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.110661:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25202688->25206783] (req 25202688->25206783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.110670:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25202688->25206783] (req 25202688->25206783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.110678:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cec00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760787.110683:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25202688->25206783] (req 25202688->25206783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.110694:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25202688->25206783] (req 25202688->25206783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.110718:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25202688->25206783] (req 25202688->25206783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.110733:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f4b964fd/0x2cf382639620a1fc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25202688->25206783] (req 25202688->25206783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.110867:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484358144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57219us (57646us total) trans 4294973471 rc 0/0 00000100:00100000:0.0:1789760787.110878:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6182 00000100:00100000:0.0:1789760787.110891:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776307579498 time 776384558218 state 0x2 ref 0 00000100:00100000:0.0:1789760787.117471:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760787.117582:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484358272 00000100:00100000:2.0:1789760787.117596:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484358272 00000100:00100000:2.0:1789760787.117678:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776307579498 time 776384558218 state 0x0 ref 1 00000100:00100000:2.0:1789760787.117699:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776384818634 time 776384558218 state 0x1 ref 1 00000100:00100000:2.0:1789760787.117796:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776384818634 time 776448725599 state 0x0 ref 1 00000100:00100000:2.0:1789760787.117812:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484358272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760787.117881:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25206784, mtime: 0x6aad9512, atime: 0x0, ctime: 0x6aad9512, blocks: 0xc807 00010000:00010000:2.0:1789760787.117892:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.117913:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:2.0:1789760787.117919:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760787.117927:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25206784->25210879] (req 25206784->25210879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.117944:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25206784->25210879] (req 25206784->25210879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760787.157103:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973472 is committed 00002000:00010000:1.0:1789760787.157194:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25206784 -> 25210880 00002000:00010000:1.0:1789760787.157199:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760786 -> 1789760787 00002000:00010000:1.0:1789760787.157201:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760786 -> 1789760787 00010000:00010000:1.0:1789760787.157217:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25206784->25210879] (req 25206784->25210879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.157238:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25206784->25210879] (req 25206784->25210879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.157248:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25206784->25210879] (req 25206784->25210879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.157257:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760787.157263:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25206784->25210879] (req 25206784->25210879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.157273:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25206784->25210879] (req 25206784->25210879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.157300:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25206784->25210879] (req 25206784->25210879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.157309:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a203 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25206784->25210879] (req 25206784->25210879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760787.157427:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98da680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484358272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39611us (39966us total) trans 4294973472 rc 0/0 00000100:00100000:1.0:1789760787.157437:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6183 00000100:00100000:1.0:1789760787.157448:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776384818634 time 776448725599 state 0x2 ref 0 00000100:00100000:1.0:1789760787.164127:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760787.164278:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484358400 00000100:00100000:3.0:1789760787.164291:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484358400 00000100:00100000:3.0:1789760787.164360:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776384818634 time 776448725599 state 0x0 ref 1 00000100:00100000:3.0:1789760787.164379:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776448986015 time 776448725599 state 0x1 ref 1 00000100:00100000:3.0:1789760787.164413:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776448986015 time 776495343630 state 0x0 ref 1 00000100:00100000:3.0:1789760787.164428:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484358400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760787.164490:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25210880, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:3.0:1789760787.164501:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.164520:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:3.0:1789760787.164524:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760787.164531:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25210880->25214975] (req 25210880->25214975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.164553:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25210880->25214975] (req 25210880->25214975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760787.216185:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973473 is committed 00002000:00010000:2.0:1789760787.216346:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25210880 -> 25214976 00010000:00010000:2.0:1789760787.216366:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25210880->25214975] (req 25210880->25214975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.216380:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25210880->25214975] (req 25210880->25214975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.216391:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25210880->25214975] (req 25210880->25214975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.216398:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760787.216404:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25210880->25214975] (req 25210880->25214975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.216414:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25210880->25214975] (req 25210880->25214975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.216439:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25210880->25214975] (req 25210880->25214975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.216448:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a20a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25210880->25214975] (req 25210880->25214975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760787.216578:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018af100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484358400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52146us (52458us total) trans 4294973473 rc 0/0 00000100:00100000:2.0:1789760787.216594:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6184 00000100:00100000:2.0:1789760787.216608:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776448986015 time 776495343630 state 0x2 ref 0 00000100:00100000:2.0:1789760787.220447:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760787.220585:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484358528 00000100:00100000:1.0:1789760787.220599:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484358528 00000100:00100000:1.0:1789760787.220677:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776448986015 time 776495343630 state 0x0 ref 1 00000100:00100000:1.0:1789760787.220696:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776495604046 time 776495343630 state 0x1 ref 1 00000100:00100000:1.0:1789760787.220881:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776495604046 time 776551809591 state 0x0 ref 1 00000100:00100000:1.0:1789760787.220897:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484358528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760787.222165:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25214976, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:1.0:1789760787.222184:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.222206:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:1.0:1789760787.222212:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760787.222220:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25214976->25219071] (req 25214976->25219071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.222238:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25214976->25219071] (req 25214976->25219071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760787.270521:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25214976 -> 25219072 00010000:00010000:1.0:1789760787.270546:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25214976->25219071] (req 25214976->25219071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.270561:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25214976->25219071] (req 25214976->25219071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.270572:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25214976->25219071] (req 25214976->25219071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.270582:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760787.270587:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25214976->25219071] (req 25214976->25219071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.270597:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25214976->25219071] (req 25214976->25219071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.270622:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25214976->25219071] (req 25214976->25219071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.270631:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a211 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25214976->25219071] (req 25214976->25219071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760787.270764:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484358528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49853us (50315us total) trans 4294973474 rc 0/0 00000100:00100000:1.0:1789760787.270776:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6185 00000100:00100000:1.0:1789760787.270788:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776495604046 time 776551809591 state 0x2 ref 0 00000001:00080000:1.0:1789760787.270843:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973474 is committed 00000100:00100000:1.0:1789760787.275028:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760787.277445:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484358656 00000100:00100000:0.0:1789760787.277471:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484358656 00000100:00100000:0.0:1789760787.277567:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776495604046 time 776551809591 state 0x0 ref 1 00000100:00100000:0.0:1789760787.277586:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776552070007 time 776551809591 state 0x1 ref 1 00000100:00100000:0.0:1789760787.277629:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776552070007 time 776608559680 state 0x0 ref 1 00000100:00100000:0.0:1789760787.277646:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484358656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760787.277726:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25219072, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:0.0:1789760787.277737:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.277758:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:0.0:1789760787.277763:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760787.277772:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25219072->25223167] (req 25219072->25223167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.277790:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25219072->25223167] (req 25219072->25223167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760787.323206:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25219072 -> 25223168 00010000:00010000:0.0:1789760787.323234:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25219072->25223167] (req 25219072->25223167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.323249:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25219072->25223167] (req 25219072->25223167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.323260:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25219072->25223167] (req 25219072->25223167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.323268:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760787.323274:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25219072->25223167] (req 25219072->25223167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.323285:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25219072->25223167] (req 25219072->25223167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.323310:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25219072->25223167] (req 25219072->25223167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.323319:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a218 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25219072->25223167] (req 25219072->25223167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.323447:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484358656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45798us (48427us total) trans 4294973475 rc 0/0 00000100:00100000:0.0:1789760787.323460:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6186 00000100:00100000:0.0:1789760787.323472:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776552070007 time 776608559680 state 0x2 ref 0 00000001:00080000:1.0:1789760787.323560:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973475 is committed 00000100:00100000:2.0:1789760787.326640:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760787.327198:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484358784 00000100:00100000:1.0:1789760787.327212:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484358784 00000100:00100000:1.0:1789760787.327286:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776552070007 time 776608559680 state 0x0 ref 1 00000100:00100000:1.0:1789760787.327308:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776608820096 time 776608559680 state 0x1 ref 1 00000100:00100000:1.0:1789760787.327376:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776608820096 time 776658305472 state 0x0 ref 1 00000100:00100000:1.0:1789760787.327391:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484358784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760787.327455:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25223168, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:1.0:1789760787.327466:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.327485:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:1.0:1789760787.327489:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760787.327496:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25223168->25227263] (req 25223168->25227263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.327513:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25223168->25227263] (req 25223168->25227263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.340379:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760787.366343:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973476 is committed 00002000:00010000:1.0:1789760787.366475:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25223168 -> 25227264 00010000:00010000:1.0:1789760787.366496:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25223168->25227263] (req 25223168->25227263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.366511:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25223168->25227263] (req 25223168->25227263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.366526:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25223168->25227263] (req 25223168->25227263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.366534:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760787.366540:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25223168->25227263] (req 25223168->25227263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.366552:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25223168->25227263] (req 25223168->25227263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.366579:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25223168->25227263] (req 25223168->25227263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.366589:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a21f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25223168->25227263] (req 25223168->25227263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760787.366707:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484358784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39311us (40074us total) trans 4294973476 rc 0/0 00000100:00100000:1.0:1789760787.366718:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6187 00000100:00100000:1.0:1789760787.366730:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776608820096 time 776658305472 state 0x2 ref 0 00000100:00100000:2.0:1789760787.376295:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760787.376397:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484358912 00000100:00100000:3.0:1789760787.376411:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484358912 00000100:00100000:3.0:1789760787.376488:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776608820096 time 776658305472 state 0x0 ref 1 00000100:00100000:3.0:1789760787.376507:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776658565888 time 776658305472 state 0x1 ref 1 00000100:00100000:3.0:1789760787.376579:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776658565888 time 776707508931 state 0x0 ref 1 00000100:00100000:3.0:1789760787.376594:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484358912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760787.376659:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25227264, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:3.0:1789760787.376669:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.376689:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:3.0:1789760787.376694:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760787.376702:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25227264->25231359] (req 25227264->25231359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.376718:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25227264->25231359] (req 25227264->25231359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760787.411521:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973477 is committed 00002000:00010000:3.0:1789760787.411700:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25227264 -> 25231360 00010000:00010000:3.0:1789760787.411720:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25227264->25231359] (req 25227264->25231359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.411735:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25227264->25231359] (req 25227264->25231359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.411746:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25227264->25231359] (req 25227264->25231359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.411878:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760787.411885:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25227264->25231359] (req 25227264->25231359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.411895:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25227264->25231359] (req 25227264->25231359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.411919:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25227264->25231359] (req 25227264->25231359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.411927:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a226 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25227264->25231359] (req 25227264->25231359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760787.412266:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484358912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35666us (35978us total) trans 4294973477 rc 0/0 00000100:00100000:3.0:1789760787.412279:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6188 00000100:00100000:3.0:1789760787.412291:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776658565888 time 776707508931 state 0x2 ref 0 00000100:00100000:2.0:1789760787.417270:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760787.417386:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484359040 00000100:00100000:0.0:1789760787.417398:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484359040 00000100:00100000:0.0:1789760787.417479:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776658565888 time 776707508931 state 0x0 ref 1 00000100:00100000:0.0:1789760787.417499:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776707769347 time 776707508931 state 0x1 ref 1 00000100:00100000:0.0:1789760787.417568:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776707769347 time 776748497624 state 0x0 ref 1 00000100:00100000:0.0:1789760787.417585:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484359040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760787.417650:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25231360, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:0.0:1789760787.417670:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.417691:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:0.0:1789760787.417696:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760787.417704:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25231360->25235455] (req 25231360->25235455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.417721:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25231360->25235455] (req 25231360->25235455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760787.449096:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973478 is committed 00002000:00010000:0.0:1789760787.449264:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25231360 -> 25235456 00010000:00010000:0.0:1789760787.449283:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25231360->25235455] (req 25231360->25235455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.449298:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25231360->25235455] (req 25231360->25235455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.449310:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25231360->25235455] (req 25231360->25235455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.449318:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760787.449324:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25231360->25235455] (req 25231360->25235455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.449334:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25231360->25235455] (req 25231360->25235455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.449359:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25231360->25235455] (req 25231360->25235455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.449367:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a22d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25231360->25235455] (req 25231360->25235455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.449493:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484359040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31905us (32428us total) trans 4294973478 rc 0/0 00000100:00100000:0.0:1789760787.449510:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6189 00000100:00100000:0.0:1789760787.449523:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776707769347 time 776748497624 state 0x2 ref 0 00000100:00100000:2.0:1789760787.452329:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760787.452413:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484359168 00000100:00100000:0.0:1789760787.452425:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484359168 00000100:00100000:0.0:1789760787.452502:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776707769347 time 776748497624 state 0x0 ref 1 00000100:00100000:0.0:1789760787.452521:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776748758040 time 776748497624 state 0x1 ref 1 00000100:00100000:0.0:1789760787.452553:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776748758040 time 776783484005 state 0x0 ref 1 00000100:00100000:0.0:1789760787.452569:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484359168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760787.452634:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25235456, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:0.0:1789760787.452645:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.452664:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:0.0:1789760787.452669:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760787.452676:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25235456->25239551] (req 25235456->25239551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.452692:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25235456->25239551] (req 25235456->25239551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760787.498709:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973479 is committed 00002000:00010000:0.0:1789760787.498915:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25235456 -> 25239552 00010000:00010000:0.0:1789760787.498938:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25235456->25239551] (req 25235456->25239551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.498953:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25235456->25239551] (req 25235456->25239551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.498963:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25235456->25239551] (req 25235456->25239551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.498980:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760787.498985:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25235456->25239551] (req 25235456->25239551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.498996:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25235456->25239551] (req 25235456->25239551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.499019:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25235456->25239551] (req 25235456->25239551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.499028:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a234 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25235456->25239551] (req 25235456->25239551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.499305:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484359168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46732us (46982us total) trans 4294973479 rc 0/0 00000100:00100000:0.0:1789760787.499318:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6190 00000100:00100000:0.0:1789760787.499331:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776748758040 time 776783484005 state 0x2 ref 0 00000100:00100000:2.0:1789760787.502369:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760787.502486:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484359296 00000100:00100000:1.0:1789760787.502500:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484359296 00000100:00100000:1.0:1789760787.502573:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776748758040 time 776783484005 state 0x0 ref 1 00000100:00100000:1.0:1789760787.502594:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776783744421 time 776783484005 state 0x1 ref 1 00000100:00100000:1.0:1789760787.502662:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776783744421 time 776833591979 state 0x0 ref 1 00000100:00100000:1.0:1789760787.502677:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484359296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760787.502744:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25239552, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:1.0:1789760787.502836:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.502856:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:1.0:1789760787.502862:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760787.502869:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25239552->25243647] (req 25239552->25243647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.502888:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25239552->25243647] (req 25239552->25243647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760787.544358:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973480 is committed 00002000:00010000:2.0:1789760787.544476:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25239552 -> 25243648 00010000:00010000:2.0:1789760787.544495:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25239552->25243647] (req 25239552->25243647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.544510:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25239552->25243647] (req 25239552->25243647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.544521:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25239552->25243647] (req 25239552->25243647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.544528:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760787.544534:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25239552->25243647] (req 25239552->25243647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.544543:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25239552->25243647] (req 25239552->25243647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.544568:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25239552->25243647] (req 25239552->25243647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.544577:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a23b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25239552->25243647] (req 25239552->25243647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760787.544689:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484359296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42009us (42329us total) trans 4294973480 rc 0/0 00000100:00100000:2.0:1789760787.544700:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6191 00000100:00100000:2.0:1789760787.544711:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776783744421 time 776833591979 state 0x2 ref 0 00000100:00100000:1.0:1789760787.548574:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760787.548670:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484359424 00000100:00100000:3.0:1789760787.548682:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484359424 00000100:00100000:3.0:1789760787.548812:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776783744421 time 776833591979 state 0x0 ref 1 00000100:00100000:3.0:1789760787.548833:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776833852395 time 776833591979 state 0x1 ref 1 00000100:00100000:3.0:1789760787.548906:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776833852395 time 776879836108 state 0x0 ref 1 00000100:00100000:3.0:1789760787.548922:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484359424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760787.548994:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25243648, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:3.0:1789760787.549005:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.549024:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:3.0:1789760787.549028:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760787.549036:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25243648->25247743] (req 25243648->25247743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.549052:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25243648->25247743] (req 25243648->25247743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760787.583611:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25243648 -> 25247744 00010000:00010000:3.0:1789760787.583637:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25243648->25247743] (req 25243648->25247743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.583660:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25243648->25247743] (req 25243648->25247743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.583670:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25243648->25247743] (req 25243648->25247743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.583678:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760787.583684:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25243648->25247743] (req 25243648->25247743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.583695:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25243648->25247743] (req 25243648->25247743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.583718:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25243648->25247743] (req 25243648->25247743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760787.583728:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a242 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25243648->25247743] (req 25243648->25247743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760787.583909:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ae300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484359424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34982us (35341us total) trans 4294973481 rc 0/0 00000100:00100000:3.0:1789760787.583922:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6192 00000100:00100000:3.0:1789760787.583934:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776833852395 time 776879836108 state 0x2 ref 0 00000001:00080000:1.0:1789760787.587653:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973481 is committed 00000100:00100000:2.0:1789760787.588336:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760787.588460:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484359552 00000100:00100000:0.0:1789760787.588472:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484359552 00000100:00100000:0.0:1789760787.588548:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776833852395 time 776879836108 state 0x0 ref 1 00000100:00100000:0.0:1789760787.588567:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776880096524 time 776879836108 state 0x1 ref 1 00000100:00100000:0.0:1789760787.588636:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776880096524 time 776919565735 state 0x0 ref 1 00000100:00100000:0.0:1789760787.588652:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484359552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760787.588724:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25247744, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:0.0:1789760787.588734:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.588762:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:0.0:1789760787.588768:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760787.588775:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25247744->25251839] (req 25247744->25251839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.588792:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25247744->25251839] (req 25247744->25251839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760787.624457:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973482 is committed 00002000:00010000:0.0:1789760787.624638:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25247744 -> 25251840 00010000:00010000:0.0:1789760787.624658:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25247744->25251839] (req 25247744->25251839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.624951:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25247744->25251839] (req 25247744->25251839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.624977:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25247744->25251839] (req 25247744->25251839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.624986:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760787.624991:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25247744->25251839] (req 25247744->25251839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.625001:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25247744->25251839] (req 25247744->25251839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.625027:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25247744->25251839] (req 25247744->25251839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.625035:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a249 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25247744->25251839] (req 25247744->25251839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.625393:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484359552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36736us (37056us total) trans 4294973482 rc 0/0 00000100:00100000:0.0:1789760787.625405:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6193 00000100:00100000:0.0:1789760787.625417:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776880096524 time 776919565735 state 0x2 ref 0 00000100:00100000:2.0:1789760787.628499:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760787.628595:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484359680 00000100:00100000:1.0:1789760787.628608:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484359680 00000100:00100000:1.0:1789760787.628675:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776880096524 time 776919565735 state 0x0 ref 1 00000100:00100000:1.0:1789760787.628693:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776919826151 time 776919565735 state 0x1 ref 1 00000100:00100000:1.0:1789760787.628726:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776919826151 time 776959655955 state 0x0 ref 1 00000100:00100000:1.0:1789760787.628741:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484359680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760787.628814:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25251840, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:1.0:1789760787.628824:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.628844:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=776s v=(0 0) 00010000:00010000:1.0:1789760787.628849:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760787.628857:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25251840->25255935] (req 25251840->25255935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.628873:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25251840->25255935] (req 25251840->25255935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760787.665553:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973483 is committed 00002000:00010000:0.0:1789760787.665735:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25251840 -> 25255936 00010000:00010000:0.0:1789760787.665768:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25251840->25255935] (req 25251840->25255935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.665789:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25251840->25255935] (req 25251840->25255935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.665799:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25251840->25255935] (req 25251840->25255935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.665808:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760787.665812:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25251840->25255935] (req 25251840->25255935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.665822:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25251840->25255935] (req 25251840->25255935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.665846:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25251840->25255935] (req 25251840->25255935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.665854:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a250 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25251840->25255935] (req 25251840->25255935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.665985:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f47b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484359680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37232us (37485us total) trans 4294973483 rc 0/0 00000100:00100000:0.0:1789760787.665998:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6194 00000100:00100000:0.0:1789760787.666010:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776919826151 time 776959655955 state 0x2 ref 0 00000100:00100000:2.0:1789760787.670246:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760787.671313:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484359808 00000100:00100000:1.0:1789760787.671331:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484359808 00000100:00100000:1.0:1789760787.671408:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776919826151 time 776959655955 state 0x0 ref 1 00000100:00100000:1.0:1789760787.671426:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776959916371 time 776959655955 state 0x1 ref 1 00000100:00100000:1.0:1789760787.671506:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776959916371 time 777002435592 state 0x0 ref 1 00000100:00100000:1.0:1789760787.671522:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484359808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760787.671590:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25255936, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:1.0:1789760787.671601:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.671630:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:1.0:1789760787.671636:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760787.671643:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25255936->25260031] (req 25255936->25260031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.671664:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25255936->25260031] (req 25255936->25260031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760787.711880:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973484 is committed 00002000:00010000:1.0:1789760787.712053:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25255936 -> 25260032 00010000:00010000:1.0:1789760787.712101:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25255936->25260031] (req 25255936->25260031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.712117:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25255936->25260031] (req 25255936->25260031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.712129:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25255936->25260031] (req 25255936->25260031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.712138:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760787.712143:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25255936->25260031] (req 25255936->25260031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.712154:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25255936->25260031] (req 25255936->25260031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.712181:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25255936->25260031] (req 25255936->25260031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.712191:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a257 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25255936->25260031] (req 25255936->25260031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760787.712317:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484359808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40792us (42079us total) trans 4294973484 rc 0/0 00000100:00100000:1.0:1789760787.712329:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6195 00000100:00100000:1.0:1789760787.712342:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776959916371 time 777002435592 state 0x2 ref 0 00000100:00100000:1.0:1789760787.715671:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760787.716038:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484359936 00000100:00100000:3.0:1789760787.716051:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484359936 00000100:00100000:2.0:1789760787.716424:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 776959916371 time 777002435592 state 0x0 ref 1 00000100:00100000:2.0:1789760787.716444:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777002696008 time 777002435592 state 0x1 ref 1 00000100:00100000:2.0:1789760787.716479:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777002696008 time 777047409444 state 0x0 ref 1 00000100:00100000:2.0:1789760787.716494:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484359936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760787.716561:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25260032, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:2.0:1789760787.716572:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.716599:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:2.0:1789760787.716604:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760787.716612:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25260032->25264127] (req 25260032->25264127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.716628:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25260032->25264127] (req 25260032->25264127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760787.746893:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973485 is committed 00002000:00010000:1.0:1789760787.747403:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25260032 -> 25264128 00010000:00010000:1.0:1789760787.747422:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25260032->25264127] (req 25260032->25264127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.747437:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25260032->25264127] (req 25260032->25264127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.747448:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25260032->25264127] (req 25260032->25264127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.747455:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760787.747461:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25260032->25264127] (req 25260032->25264127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.747472:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25260032->25264127] (req 25260032->25264127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.747501:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25260032->25264127] (req 25260032->25264127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.747509:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a25e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25260032->25264127] (req 25260032->25264127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760787.747618:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf5018ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484359936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31121us (31954us total) trans 4294973485 rc 0/0 00000100:00100000:1.0:1789760787.747630:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6196 00000100:00100000:1.0:1789760787.747642:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777002696008 time 777047409444 state 0x2 ref 0 00000100:00100000:0.0:1789760787.750681:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760787.750777:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484360064 00000100:00100000:2.0:1789760787.750788:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484360064 00000100:00100000:2.0:1789760787.750850:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777002696008 time 777047409444 state 0x0 ref 1 00000100:00100000:2.0:1789760787.750866:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777047669860 time 777047409444 state 0x1 ref 1 00000100:00100000:2.0:1789760787.750926:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777047669860 time 777081855617 state 0x0 ref 1 00000100:00100000:2.0:1789760787.750941:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484360064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760787.751020:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25264128, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:2.0:1789760787.751031:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.751050:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:2.0:1789760787.751055:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760787.751061:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25264128->25268223] (req 25264128->25268223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760787.751102:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25264128->25268223] (req 25264128->25268223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760787.792579:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25264128 -> 25268224 00010000:00010000:0.0:1789760787.792674:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25264128->25268223] (req 25264128->25268223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.792693:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25264128->25268223] (req 25264128->25268223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.792704:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25264128->25268223] (req 25264128->25268223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.792712:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760787.792718:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25264128->25268223] (req 25264128->25268223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.792729:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25264128->25268223] (req 25264128->25268223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.792766:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25264128->25268223] (req 25264128->25268223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.792774:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a265 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25264128->25268223] (req 25264128->25268223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.792963:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484360064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42017us (42285us total) trans 4294973486 rc 0/0 00000100:00100000:0.0:1789760787.792986:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6197 00000100:00100000:0.0:1789760787.793000:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777047669860 time 777081855617 state 0x2 ref 0 00000001:00080000:0.0:1789760787.793100:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973486 is committed 00000100:00100000:2.0:1789760787.801334:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760787.801619:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484360192 00000100:00100000:0.0:1789760787.801632:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484360192 00000100:00100000:0.0:1789760787.801725:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777047669860 time 777081855617 state 0x0 ref 1 00000100:00100000:0.0:1789760787.801745:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777082116033 time 777081855617 state 0x1 ref 1 00000100:00100000:0.0:1789760787.801784:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777082116033 time 777132714729 state 0x0 ref 1 00000100:00100000:0.0:1789760787.801800:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484360192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760787.801867:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25268224, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:0.0:1789760787.801878:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.801900:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:0.0:1789760787.801917:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760787.801925:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25268224->25272319] (req 25268224->25272319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.801943:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25268224->25272319] (req 25268224->25272319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.852309:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508941440:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760787.852412:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.852614:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508941440 00000100:00100000:1.0:1789760787.852623:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508941440 00000100:00100000:1.0:1789760787.852685:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 416 00000100:00100000:1.0:1789760787.852695:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508941440:12345-0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760787.852748:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8abe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508941568:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1789760787.852787:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760787.852822:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508941568 00000100:00100000:0.0:1789760787.852827:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508941568 00000100:00100000:0.0:1789760787.852852:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 417 00000100:00100000:0.0:1789760787.852860:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508941568:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:1.0:1789760787.852914:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1789760787.852979:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508941440:12345-0@lo:13:osp-pre-1-1.0 Request processed in 275us (556us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760787.852988:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 416 00000100:00080000:1.0:1789760787.853016:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:1.0:1789760787.853059:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98dad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508941440:0@lo:13:osp-pre-1-1.0 00000100:00100000:1.0:1789760787.853448:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f46300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3479:1876699508941696:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.853485:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.853517:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba74700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508942208:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.853542:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.853558:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508942720:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.853577:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.853595:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1789760787.853600:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508942080:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.853622:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.853638:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508942592:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.853654:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.853676:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508942208 00000100:00100000:1.0:1789760787.853680:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508942208 00000100:00100000:1.0:1789760787.853709:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 475 00000100:00100000:1.0:1789760787.853717:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508942208:12345-0@lo:400:kworker.0 00002000:00000024:0.0:1789760787.853885:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:0.0:1789760787.853930:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98db480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508941568:12345-0@lo:13:osp-pre-1-0.0 Request processed in 1070us (1140us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760787.853938:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 417 00000100:00100000:1.0:1789760787.853944:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508942208:12345-0@lo:400:kworker.0 Request processed in 225us (399us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760787.853952:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 475 00000100:00100000:1.0:1789760787.853987:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508942720 00000100:00100000:1.0:1789760787.853991:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508942720 00000100:00100000:1.0:1789760787.854025:0:8101:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508942592 00000100:00100000:1.0:1789760787.854028:0:8101:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508942592 00000100:00100000:1.0:1789760787.854056:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 351 00000100:00100000:1.0:1789760787.854081:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508942720:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854113:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508942720:12345-0@lo:400:kworker.0 Request processed in 50us (534us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760787.854120:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 351 00000100:00100000:1.0:1789760787.854133:0:8101:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 352 00000100:00100000:1.0:1789760787.854141:0:8101:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508942592:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854172:0:8101:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508942592:12345-0@lo:400:kworker.0 Request processed in 33us (517us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760787.854179:0:8101:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 352 00000100:00100000:2.0:1789760787.854187:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f44000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508941824:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854200:0:6475:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508942080 00000100:00100000:1.0:1789760787.854204:0:6475:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508942080 00000100:00100000:1.0:1789760787.854229:0:6475:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1005 00000100:00100000:1.0:1789760787.854235:0:6475:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508942080:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854261:0:6475:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0001_UUID+5:3479:x1876699508942080:12345-0@lo:400:kworker.0 Request processed in 26us (638us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760787.854266:0:6475:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1005 00000100:00100000:1.0:1789760787.854320:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0001_UUID:3479:1876699508942080:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854330:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508942592:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854339:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba74700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508942208:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854347:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508942720:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854374:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508941952:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854398:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.854414:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0000_UUID:3481:1876699508942464:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854429:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.854444:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c457bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0001_UUID:3481:1876699508942976:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854459:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.854484:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508941952 00000100:00100000:1.0:1789760787.854487:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508941952 00000100:00100000:2.0:1789760787.854506:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760787.854510:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 476 00000100:00100000:1.0:1789760787.854517:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3481:x1876699508941952:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854553:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc7b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3481:x1876699508941952:12345-0@lo:400:kworker.0 Request processed in 36us (154us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760787.854559:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 476 00000100:00100000:1.0:1789760787.854572:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508942464 00000100:00100000:1.0:1789760787.854575:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508942464 00000100:00100000:1.0:1789760787.854593:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508942976 00000100:00100000:2.0:1789760787.854595:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508942336:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854596:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508942976 00000100:00100000:1.0:1789760787.854612:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508941824 00000100:00100000:1.0:1789760787.854616:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508941824 00000100:00100000:1.0:1789760787.854630:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1006 00000100:00100000:1.0:1789760787.854635:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0000_UUID+5:3481:x1876699508942464:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854659:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0000_UUID+5:3481:x1876699508942464:12345-0@lo:400:kworker.0 Request processed in 25us (229us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760787.854665:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1006 00000100:00100000:1.0:1789760787.854673:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1007 00000100:00100000:1.0:1789760787.854677:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0001_UUID+5:3481:x1876699508942976:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.854705:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc4e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0001_UUID+5:3481:x1876699508942976:12345-0@lo:400:kworker.0 Request processed in 27us (245us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760787.854711:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1007 00000100:00100000:1.0:1789760787.854724:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1008 00000100:00100000:1.0:1789760787.854729:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508941824:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760787.854814:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760787.854831:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c4578e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508942848:0@lo:400:kworker.0 00000100:00100000:2.0:1789760787.854854:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760787.854879:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508942336 00000100:00100000:2.0:1789760787.854883:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508942336 00000100:00100000:2.0:1789760787.854904:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508942848 00000100:00100000:2.0:1789760787.854907:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508942848 00000100:00100000:2.0:1789760787.854925:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1009 00000100:00100000:2.0:1789760787.854933:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508942336:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760787.854951:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508941696 00000100:00100000:3.0:1789760787.854957:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508941696 00000100:00100000:2.0:1789760787.854988:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508942336:12345-0@lo:400:kworker.0 Request processed in 55us (174us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760787.854996:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1009 00000100:00100000:3.0:1789760787.855007:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 496 00000100:00100000:2.0:1789760787.855015:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1010 00000100:00100000:3.0:1789760787.855021:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c803bc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3479:x1876699508941696:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760787.855037:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508942848:12345-0@lo:400:kworker.0 00000100:00080000:0.0:1789760787.856045:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:3.0:1789760787.856061:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c803bc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3479:x1876699508941696:12345-0@lo:400:kworker.0 Request processed in 1042us (2570us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760787.856088:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 496 00000100:00080000:0.0:1789760787.856276:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:0.0:1789760787.856283:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1789760787.856344:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508941824:12345-0@lo:400:kworker.0 Request processed in 1613us (1838us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760787.856352:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1008 00000100:00100000:1.0:1789760787.856419:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f46300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3479:1876699508941696:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.856455:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508941952:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.856467:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0000_UUID:3481:1876699508942464:0@lo:400:kworker.0 00000100:00100000:1.0:1789760787.856481:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c457bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0001_UUID:3481:1876699508942976:0@lo:400:kworker.0 00000100:00100000:2.0:1789760787.856680:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508942848:12345-0@lo:400:kworker.0 Request processed in 1655us (1823us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760787.856688:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1010 00000100:00100000:2.0:1789760787.858236:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8abe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508941568:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760787.858290:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f44000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508941824:0@lo:400:kworker.0 00000100:00100000:2.0:1789760787.858305:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508942336:0@lo:400:kworker.0 00000100:00100000:2.0:1789760787.858318:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c4578e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508942848:0@lo:400:kworker.0 00000001:00080000:2.0:1789760787.861595:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973487 is committed 00002000:00010000:1.0:1789760787.862707:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25268224 -> 25272320 00010000:00010000:1.0:1789760787.862728:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25268224->25272319] (req 25268224->25272319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.862986:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25268224->25272319] (req 25268224->25272319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.862996:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25268224->25272319] (req 25268224->25272319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.863004:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760787.863009:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25268224->25272319] (req 25268224->25272319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.863089:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25268224->25272319] (req 25268224->25272319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.863115:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25268224->25272319] (req 25268224->25272319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.863124:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a26c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25268224->25272319] (req 25268224->25272319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.863536:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484360192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61731us (62209us total) trans 4294973487 rc 0/0 00000100:00100000:0.0:1789760787.863551:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6198 00000100:00100000:0.0:1789760787.863563:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777082116033 time 777132714729 state 0x2 ref 0 00000100:00100000:2.0:1789760787.866577:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760787.866649:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484360320 00000100:00100000:1.0:1789760787.866667:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484360320 00000100:00100000:1.0:1789760787.866743:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777082116033 time 777132714729 state 0x0 ref 1 00000100:00100000:1.0:1789760787.866773:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777132975145 time 777132714729 state 0x1 ref 1 00000100:00100000:1.0:1789760787.866809:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777132975145 time 777197738898 state 0x0 ref 1 00000100:00100000:1.0:1789760787.866823:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484360320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760787.866889:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25272320, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:1.0:1789760787.866900:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.866919:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:1.0:1789760787.866924:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760787.866933:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25272320->25276415] (req 25272320->25276415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.866948:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25272320->25276415] (req 25272320->25276415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760787.906041:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973488 is committed 00002000:00010000:1.0:1789760787.906250:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25272320 -> 25276416 00010000:00010000:1.0:1789760787.906271:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25272320->25276415] (req 25272320->25276415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.906286:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25272320->25276415] (req 25272320->25276415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.906302:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25272320->25276415] (req 25272320->25276415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.906310:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760787.906315:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25272320->25276415] (req 25272320->25276415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.906325:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25272320->25276415] (req 25272320->25276415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.906352:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25272320->25276415] (req 25272320->25276415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.906361:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a273 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25272320->25276415] (req 25272320->25276415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760787.906475:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484360320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39648us (39901us total) trans 4294973488 rc 0/0 00000100:00100000:1.0:1789760787.906487:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6199 00000100:00100000:1.0:1789760787.906499:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777132975145 time 777197738898 state 0x2 ref 0 00000100:00100000:2.0:1789760787.909447:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760787.912327:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484360448 00000100:00100000:1.0:1789760787.912352:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484360448 00000100:00100000:1.0:1789760787.912444:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777132975145 time 777197738898 state 0x0 ref 1 00000100:00100000:1.0:1789760787.912462:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777197999314 time 777197738898 state 0x1 ref 1 00000100:00100000:1.0:1789760787.912489:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777197999314 time 777243420499 state 0x0 ref 1 00000100:00100000:1.0:1789760787.912504:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484360448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760787.912572:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25276416, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:1.0:1789760787.912584:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.912604:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:1.0:1789760787.912609:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760787.912618:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25276416->25280511] (req 25276416->25280511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.912635:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25276416->25280511] (req 25276416->25280511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760787.952423:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973489 is committed 00002000:00010000:1.0:1789760787.952638:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25276416 -> 25280512 00010000:00010000:1.0:1789760787.952660:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25276416->25280511] (req 25276416->25280511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.952676:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25276416->25280511] (req 25276416->25280511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.952687:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25276416->25280511] (req 25276416->25280511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.952696:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760787.952702:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25276416->25280511] (req 25276416->25280511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.952713:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25276416->25280511] (req 25276416->25280511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.952740:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25276416->25280511] (req 25276416->25280511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760787.952758:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a27a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25276416->25280511] (req 25276416->25280511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760787.952873:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484360448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40365us (43433us total) trans 4294973489 rc 0/0 00000100:00100000:1.0:1789760787.952885:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6200 00000100:00100000:1.0:1789760787.952897:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777197999314 time 777243420499 state 0x2 ref 0 00000100:00100000:2.0:1789760787.955678:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760787.955812:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484360576 00000100:00100000:0.0:1789760787.955825:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484360576 00000100:00100000:0.0:1789760787.955897:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777197999314 time 777243420499 state 0x0 ref 1 00000100:00100000:0.0:1789760787.955915:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777243680915 time 777243420499 state 0x1 ref 1 00000100:00100000:0.0:1789760787.955987:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777243680915 time 777286915538 state 0x0 ref 1 00000100:00100000:0.0:1789760787.956003:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484360576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760787.956151:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25280512, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:0.0:1789760787.956162:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.956181:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:0.0:1789760787.956186:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760787.956194:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25280512->25284607] (req 25280512->25284607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.956210:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25280512->25284607] (req 25280512->25284607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760787.997808:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973490 is committed 00002000:00010000:0.0:1789760787.997946:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25280512 -> 25284608 00010000:00010000:0.0:1789760787.997975:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25280512->25284607] (req 25280512->25284607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.997992:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25280512->25284607] (req 25280512->25284607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.998001:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25280512->25284607] (req 25280512->25284607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.998008:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760787.998014:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25280512->25284607] (req 25280512->25284607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.998023:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25280512->25284607] (req 25280512->25284607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.998048:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25280512->25284607] (req 25280512->25284607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760787.998057:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a281 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25280512->25284607] (req 25280512->25284607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760787.998198:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484360576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42192us (42528us total) trans 4294973490 rc 0/0 00000100:00100000:0.0:1789760787.998209:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6201 00000100:00100000:0.0:1789760787.998222:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777243680915 time 777286915538 state 0x2 ref 0 00000100:00100000:2.0:1789760788.001831:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760788.002473:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484360704 00000100:00100000:0.0:1789760788.002487:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484360704 00000100:00100000:0.0:1789760788.002630:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777243680915 time 777286915538 state 0x0 ref 1 00000100:00100000:0.0:1789760788.002648:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777287175954 time 777286915538 state 0x1 ref 1 00000100:00100000:0.0:1789760788.002684:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777287175954 time 777333614520 state 0x0 ref 1 00000100:00100000:0.0:1789760788.002706:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484360704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760788.002768:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25284608, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:0.0:1789760788.002777:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.002797:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:0.0:1789760788.002803:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760788.002811:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25284608->25288703] (req 25284608->25288703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.002829:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25284608->25288703] (req 25284608->25288703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1789760788.003666:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760788.041056:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973491 is committed 00002000:00010000:0.0:1789760788.041234:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25284608 -> 25288704 00010000:00010000:0.0:1789760788.041254:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25284608->25288703] (req 25284608->25288703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.041267:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25284608->25288703] (req 25284608->25288703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.041277:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25284608->25288703] (req 25284608->25288703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.041354:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760788.041361:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25284608->25288703] (req 25284608->25288703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.041373:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25284608->25288703] (req 25284608->25288703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.041399:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25284608->25288703] (req 25284608->25288703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.041413:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a288 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25284608->25288703] (req 25284608->25288703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760788.041583:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484360704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38879us (39758us total) trans 4294973491 rc 0/0 00000100:00100000:0.0:1789760788.041595:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6202 00000100:00100000:0.0:1789760788.041609:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777287175954 time 777333614520 state 0x2 ref 0 00000100:00100000:3.0:1789760788.049824:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760788.049933:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484360832 00000100:00100000:1.0:1789760788.049946:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484360832 00000100:00100000:1.0:1789760788.050028:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777287175954 time 777333614520 state 0x0 ref 1 00000100:00100000:1.0:1789760788.050047:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777333874936 time 777333614520 state 0x1 ref 1 00000100:00100000:1.0:1789760788.050129:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777333874936 time 777381058530 state 0x0 ref 1 00000100:00100000:1.0:1789760788.050144:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484360832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760788.050217:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25288704, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:1.0:1789760788.050229:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.050251:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:1.0:1789760788.050257:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760788.050265:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25288704->25292799] (req 25288704->25292799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.050281:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25288704->25292799] (req 25288704->25292799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760788.089719:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973492 is committed 00002000:00010000:1.0:1789760788.090100:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25288704 -> 25292800 00010000:00010000:1.0:1789760788.090122:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25288704->25292799] (req 25288704->25292799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.090137:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25288704->25292799] (req 25288704->25292799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.090148:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25288704->25292799] (req 25288704->25292799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.090156:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760788.090162:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25288704->25292799] (req 25288704->25292799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.090172:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25288704->25292799] (req 25288704->25292799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.090198:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25288704->25292799] (req 25288704->25292799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.090206:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a28f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25288704->25292799] (req 25288704->25292799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760788.090395:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484360832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40245us (40576us total) trans 4294973492 rc 0/0 00000100:00100000:1.0:1789760788.090407:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6203 00000100:00100000:1.0:1789760788.090419:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777333874936 time 777381058530 state 0x2 ref 0 00000100:00100000:2.0:1789760788.095522:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760788.095643:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484360960 00000100:00100000:1.0:1789760788.095659:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484360960 00000100:00100000:1.0:1789760788.095737:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777333874936 time 777381058530 state 0x0 ref 1 00000100:00100000:1.0:1789760788.095767:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777381318946 time 777381058530 state 0x1 ref 1 00000100:00100000:1.0:1789760788.095802:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777381318946 time 777426732147 state 0x0 ref 1 00000100:00100000:1.0:1789760788.095816:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484360960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760788.095883:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25292800, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:1.0:1789760788.095894:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.096173:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:1.0:1789760788.096180:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760788.096189:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25292800->25296895] (req 25292800->25296895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.096208:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25292800->25296895] (req 25292800->25296895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760788.148540:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973493 is committed 00002000:00010000:2.0:1789760788.148688:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25292800 -> 25296896 00010000:00010000:2.0:1789760788.148708:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25292800->25296895] (req 25292800->25296895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.148724:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25292800->25296895] (req 25292800->25296895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.148734:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25292800->25296895] (req 25292800->25296895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.148742:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760788.148815:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25292800->25296895] (req 25292800->25296895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.148828:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25292800->25296895] (req 25292800->25296895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.148856:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25292800->25296895] (req 25292800->25296895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.148865:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a296 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25292800->25296895] (req 25292800->25296895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760788.149036:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abe300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484360960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53216us (53523us total) trans 4294973493 rc 0/0 00000100:00100000:2.0:1789760788.149049:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6204 00000100:00100000:2.0:1789760788.149061:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777381318946 time 777426732147 state 0x2 ref 0 00000100:00100000:0.0:1789760788.152191:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760788.152299:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484361088 00000100:00100000:2.0:1789760788.152315:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484361088 00000100:00100000:2.0:1789760788.152389:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777381318946 time 777426732147 state 0x0 ref 1 00000100:00100000:2.0:1789760788.152408:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777426992563 time 777426732147 state 0x1 ref 1 00000100:00100000:2.0:1789760788.152445:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777426992563 time 777483376362 state 0x0 ref 1 00000100:00100000:2.0:1789760788.152459:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484361088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760788.152536:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25296896, mtime: 0x6aad9513, atime: 0x0, ctime: 0x6aad9513, blocks: 0xc807 00010000:00010000:2.0:1789760788.152547:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.152569:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:2.0:1789760788.152575:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760788.152582:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25296896->25300991] (req 25296896->25300991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.152599:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25296896->25300991] (req 25296896->25300991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760788.183479:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973494 is committed 00002000:00010000:3.0:1789760788.183787:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25296896 -> 25300992 00002000:00010000:3.0:1789760788.183793:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760787 -> 1789760788 00002000:00010000:3.0:1789760788.183795:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760787 -> 1789760788 00010000:00010000:3.0:1789760788.183812:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25296896->25300991] (req 25296896->25300991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.183829:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25296896->25300991] (req 25296896->25300991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.183838:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25296896->25300991] (req 25296896->25300991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.183845:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760788.183851:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25296896->25300991] (req 25296896->25300991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.183863:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25296896->25300991] (req 25296896->25300991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.183892:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25296896->25300991] (req 25296896->25300991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.183901:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a29d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25296896->25300991] (req 25296896->25300991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760788.184058:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484361088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31594us (31875us total) trans 4294973494 rc 0/0 00000100:00100000:3.0:1789760788.184102:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6205 00000100:00100000:3.0:1789760788.184114:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777426992563 time 777483376362 state 0x2 ref 0 00000100:00100000:3.0:1789760788.191235:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760788.191579:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484361216 00000100:00100000:0.0:1789760788.191594:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484361216 00000100:00100000:0.0:1789760788.191677:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777426992563 time 777483376362 state 0x0 ref 1 00000100:00100000:0.0:1789760788.191695:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777483636778 time 777483376362 state 0x1 ref 1 00000100:00100000:0.0:1789760788.191729:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777483636778 time 777522660282 state 0x0 ref 1 00000100:00100000:0.0:1789760788.191745:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484361216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760788.191813:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25300992, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:0.0:1789760788.191824:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.191843:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:0.0:1789760788.191848:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760788.191856:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25300992->25305087] (req 25300992->25305087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.191872:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25300992->25305087] (req 25300992->25305087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760788.225850:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973495 is committed 00002000:00010000:3.0:1789760788.226134:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25300992 -> 25305088 00010000:00010000:3.0:1789760788.226155:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25300992->25305087] (req 25300992->25305087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.226171:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25300992->25305087] (req 25300992->25305087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.226181:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25300992->25305087] (req 25300992->25305087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.226193:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760788.226199:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25300992->25305087] (req 25300992->25305087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.226209:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25300992->25305087] (req 25300992->25305087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.226234:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25300992->25305087] (req 25300992->25305087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.226241:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a2a4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25300992->25305087] (req 25300992->25305087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760788.226366:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484361216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34617us (35135us total) trans 4294973495 rc 0/0 00000100:00100000:3.0:1789760788.226380:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6206 00000100:00100000:3.0:1789760788.226393:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777483636778 time 777522660282 state 0x2 ref 0 00000100:00100000:2.0:1789760788.230681:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760788.231660:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484361344 00000100:00100000:1.0:1789760788.231678:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484361344 00000100:00100000:0.0:1789760788.231866:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777483636778 time 777522660282 state 0x0 ref 1 00000100:00100000:0.0:1789760788.231886:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777522920698 time 777522660282 state 0x1 ref 1 00000100:00100000:0.0:1789760788.231921:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777522920698 time 777562851547 state 0x0 ref 1 00000100:00100000:0.0:1789760788.231937:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484361344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760788.232136:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25305088, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:0.0:1789760788.232149:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.232168:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:0.0:1789760788.232173:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760788.232180:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25305088->25309183] (req 25305088->25309183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.232197:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25305088->25309183] (req 25305088->25309183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760788.287454:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25305088 -> 25309184 00010000:00010000:0.0:1789760788.287484:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25305088->25309183] (req 25305088->25309183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.287500:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25305088->25309183] (req 25305088->25309183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.287509:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25305088->25309183] (req 25305088->25309183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.287517:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760788.287523:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25305088->25309183] (req 25305088->25309183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.287532:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25305088->25309183] (req 25305088->25309183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.287555:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25305088->25309183] (req 25305088->25309183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.287563:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a2ab lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25305088->25309183] (req 25305088->25309183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760788.287688:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484361344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55748us (57017us total) trans 4294973496 rc 0/0 00000100:00100000:0.0:1789760788.287700:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6207 00000100:00100000:0.0:1789760788.287713:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777522920698 time 777562851547 state 0x2 ref 0 00000001:00080000:1.0:1789760788.288554:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973496 is committed 00000100:00100000:0.0:1789760788.295208:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760788.295392:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484361472 00000100:00100000:2.0:1789760788.295406:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484361472 00000100:00100000:2.0:1789760788.295549:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777522920698 time 777562851547 state 0x0 ref 1 00000100:00100000:2.0:1789760788.295569:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777563111963 time 777562851547 state 0x1 ref 1 00000100:00100000:2.0:1789760788.295648:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777563111963 time 777626577156 state 0x0 ref 1 00000100:00100000:2.0:1789760788.295664:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484361472:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760788.295731:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25309184, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:2.0:1789760788.295742:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.295773:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:2.0:1789760788.295778:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760788.295787:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25309184->25313279] (req 25309184->25313279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.295804:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25309184->25313279] (req 25309184->25313279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760788.361100:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25309184 -> 25313280 00010000:00010000:1.0:1789760788.361129:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25309184->25313279] (req 25309184->25313279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.361146:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25309184->25313279] (req 25309184->25313279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.361156:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25309184->25313279] (req 25309184->25313279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.361165:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760788.361172:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25309184->25313279] (req 25309184->25313279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.361183:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25309184->25313279] (req 25309184->25313279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.361213:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25309184->25313279] (req 25309184->25313279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.361220:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a2b2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25309184->25313279] (req 25309184->25313279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760788.361556:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98daa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484361472:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65811us (66279us total) trans 4294973497 rc 0/0 00000100:00100000:1.0:1789760788.361570:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6208 00000100:00100000:1.0:1789760788.361584:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777563111963 time 777626577156 state 0x2 ref 0 00000001:00080000:0.0:1789760788.362615:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973497 is committed 00010000:00010000:1.0:1789760788.364614:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760788.372847:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760788.373011:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484361600 00000100:00100000:2.0:1789760788.373026:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484361600 00000100:00100000:2.0:1789760788.373145:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777563111963 time 777626577156 state 0x0 ref 1 00000100:00100000:2.0:1789760788.373164:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777626837572 time 777626577156 state 0x1 ref 1 00000100:00100000:2.0:1789760788.373190:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777626837572 time 777704121232 state 0x0 ref 1 00000100:00100000:2.0:1789760788.373205:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484361600:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760788.373274:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25313280, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:2.0:1789760788.373295:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.373314:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:2.0:1789760788.373320:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760788.373327:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25313280->25317375] (req 25313280->25317375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.373342:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25313280->25317375] (req 25313280->25317375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760788.424975:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973498 is committed 00002000:00010000:2.0:1789760788.425804:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25313280 -> 25317376 00010000:00010000:2.0:1789760788.425829:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25313280->25317375] (req 25313280->25317375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.425851:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25313280->25317375] (req 25313280->25317375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.425860:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25313280->25317375] (req 25313280->25317375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.425868:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760788.425874:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25313280->25317375] (req 25313280->25317375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.425885:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25313280->25317375] (req 25313280->25317375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.425912:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25313280->25317375] (req 25313280->25317375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.425920:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a2b9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25313280->25317375] (req 25313280->25317375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760788.426107:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484361600:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52852us (53223us total) trans 4294973498 rc 0/0 00000100:00100000:2.0:1789760788.426120:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6209 00000100:00100000:2.0:1789760788.426132:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777626837572 time 777704121232 state 0x2 ref 0 00000100:00100000:3.0:1789760788.431663:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760788.431743:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484361728 00000100:00100000:0.0:1789760788.431755:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484361728 00000100:00100000:0.0:1789760788.431841:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777626837572 time 777704121232 state 0x0 ref 1 00000100:00100000:0.0:1789760788.431860:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777704381648 time 777704121232 state 0x1 ref 1 00000100:00100000:0.0:1789760788.431902:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777704381648 time 777762824319 state 0x0 ref 1 00000100:00100000:0.0:1789760788.431918:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484361728:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760788.431996:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25317376, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:0.0:1789760788.432007:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.432027:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:0.0:1789760788.432032:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760788.432039:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25317376->25321471] (req 25317376->25321471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.432055:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25317376->25321471] (req 25317376->25321471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760788.464000:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973499 is committed 00002000:00010000:3.0:1789760788.464211:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25317376 -> 25321472 00010000:00010000:3.0:1789760788.464230:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25317376->25321471] (req 25317376->25321471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.464251:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25317376->25321471] (req 25317376->25321471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.464260:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25317376->25321471] (req 25317376->25321471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.464268:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760788.464274:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25317376->25321471] (req 25317376->25321471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.464352:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25317376->25321471] (req 25317376->25321471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.464380:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25317376->25321471] (req 25317376->25321471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.464389:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a2c0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25317376->25321471] (req 25317376->25321471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760788.464558:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484361728:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32636us (32901us total) trans 4294973499 rc 0/0 00000100:00100000:3.0:1789760788.464572:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6210 00000100:00100000:3.0:1789760788.464585:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777704381648 time 777762824319 state 0x2 ref 0 00000100:00100000:0.0:1789760788.469600:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760788.469723:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484361856 00000100:00100000:1.0:1789760788.469737:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484361856 00000100:00100000:1.0:1789760788.469809:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777704381648 time 777762824319 state 0x0 ref 1 00000100:00100000:1.0:1789760788.469828:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777763084735 time 777762824319 state 0x1 ref 1 00000100:00100000:1.0:1789760788.469863:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777763084735 time 777800792975 state 0x0 ref 1 00000100:00100000:1.0:1789760788.469877:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484361856:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760788.469942:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25321472, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:1.0:1789760788.469953:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.469988:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:1.0:1789760788.469993:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760788.470000:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25321472->25325567] (req 25321472->25325567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.470018:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25321472->25325567] (req 25321472->25325567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760788.505002:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973500 is committed 00002000:00010000:1.0:1789760788.505152:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25321472 -> 25325568 00010000:00010000:1.0:1789760788.505172:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25321472->25325567] (req 25321472->25325567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.505188:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25321472->25325567] (req 25321472->25325567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.505198:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25321472->25325567] (req 25321472->25325567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.505207:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760788.505212:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25321472->25325567] (req 25321472->25325567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.505222:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25321472->25325567] (req 25321472->25325567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.505249:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25321472->25325567] (req 25321472->25325567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.505259:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2c7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25321472->25325567] (req 25321472->25325567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760788.505383:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484361856:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35501us (35793us total) trans 4294973500 rc 0/0 00000100:00100000:1.0:1789760788.505395:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6211 00000100:00100000:1.0:1789760788.505406:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777763084735 time 777800792975 state 0x2 ref 0 00000100:00100000:2.0:1789760788.511340:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760788.511494:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484361984 00000100:00100000:1.0:1789760788.511509:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484361984 00000100:00100000:1.0:1789760788.511584:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777763084735 time 777800792975 state 0x0 ref 1 00000100:00100000:1.0:1789760788.511602:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777801053391 time 777800792975 state 0x1 ref 1 00000100:00100000:1.0:1789760788.511640:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777801053391 time 777842569050 state 0x0 ref 1 00000100:00100000:1.0:1789760788.511656:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484361984:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760788.511724:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25325568, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:1.0:1789760788.511736:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.511756:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:1.0:1789760788.511761:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760788.511769:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25325568->25329663] (req 25325568->25329663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.511791:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25325568->25329663] (req 25325568->25329663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760788.542916:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25325568 -> 25329664 00010000:00010000:1.0:1789760788.542947:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25325568->25329663] (req 25325568->25329663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.542970:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25325568->25329663] (req 25325568->25329663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.542981:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25325568->25329663] (req 25325568->25329663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.542989:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760788.542995:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25325568->25329663] (req 25325568->25329663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.543006:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25325568->25329663] (req 25325568->25329663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.543033:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25325568->25329663] (req 25325568->25329663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.543042:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a2ce lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25325568->25329663] (req 25325568->25329663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760788.543194:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484361984:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31533us (31862us total) trans 4294973501 rc 0/0 00000100:00100000:1.0:1789760788.543206:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6212 00000100:00100000:1.0:1789760788.543218:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777801053391 time 777842569050 state 0x2 ref 0 00000001:00080000:3.0:1789760788.543802:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973501 is committed 00000100:00100000:3.0:1789760788.546149:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760788.546920:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484362112 00000100:00100000:2.0:1789760788.546933:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484362112 00000100:00100000:2.0:1789760788.547027:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777801053391 time 777842569050 state 0x0 ref 1 00000100:00100000:2.0:1789760788.547046:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777842829466 time 777842569050 state 0x1 ref 1 00000100:00100000:2.0:1789760788.547107:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777842829466 time 777878036279 state 0x0 ref 1 00000100:00100000:2.0:1789760788.547123:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484362112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760788.547189:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25329664, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:2.0:1789760788.547201:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.547221:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:2.0:1789760788.547226:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760788.547235:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25329664->25333759] (req 25329664->25333759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.547251:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25329664->25333759] (req 25329664->25333759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760788.584851:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973502 is committed 00002000:00010000:2.0:1789760788.588992:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25329664 -> 25333760 00010000:00010000:2.0:1789760788.589022:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25329664->25333759] (req 25329664->25333759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.589038:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25329664->25333759] (req 25329664->25333759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.589055:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25329664->25333759] (req 25329664->25333759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.589085:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760788.589092:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25329664->25333759] (req 25329664->25333759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.589102:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25329664->25333759] (req 25329664->25333759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.589130:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25329664->25333759] (req 25329664->25333759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.589138:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a2d5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25329664->25333759] (req 25329664->25333759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760788.589304:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484362112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42168us (43151us total) trans 4294973502 rc 0/0 00000100:00100000:2.0:1789760788.589317:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6213 00000100:00100000:2.0:1789760788.589329:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777842829466 time 777878036279 state 0x2 ref 0 00000100:00100000:0.0:1789760788.598008:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760788.598219:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484362240 00000100:00100000:3.0:1789760788.598234:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484362240 00000100:00100000:3.0:1789760788.598308:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777842829466 time 777878036279 state 0x0 ref 1 00000100:00100000:3.0:1789760788.598326:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777878296695 time 777878036279 state 0x1 ref 1 00000100:00100000:3.0:1789760788.598363:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777878296695 time 777929293148 state 0x0 ref 1 00000100:00100000:3.0:1789760788.598378:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484362240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760788.598442:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25333760, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:3.0:1789760788.598453:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.598474:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:3.0:1789760788.598479:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760788.598487:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25333760->25337855] (req 25333760->25337855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.598503:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25333760->25337855] (req 25333760->25337855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760788.634796:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973503 is committed 00002000:00010000:3.0:1789760788.635043:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25333760 -> 25337856 00010000:00010000:3.0:1789760788.635093:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25333760->25337855] (req 25333760->25337855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.635115:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25333760->25337855] (req 25333760->25337855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.635125:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25333760->25337855] (req 25333760->25337855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.635134:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760788.635139:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25333760->25337855] (req 25333760->25337855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.635149:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25333760->25337855] (req 25333760->25337855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.635175:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25333760->25337855] (req 25333760->25337855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.635184:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2dc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25333760->25337855] (req 25333760->25337855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760788.635314:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484362240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36932us (37314us total) trans 4294973503 rc 0/0 00000100:00100000:3.0:1789760788.635327:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6214 00000100:00100000:3.0:1789760788.635339:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777878296695 time 777929293148 state 0x2 ref 0 00000100:00100000:0.0:1789760788.639317:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760788.639414:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484362368 00000100:00100000:3.0:1789760788.639429:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484362368 00000100:00100000:3.0:1789760788.639501:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777878296695 time 777929293148 state 0x0 ref 1 00000100:00100000:3.0:1789760788.639522:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777929553564 time 777929293148 state 0x1 ref 1 00000100:00100000:3.0:1789760788.639555:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777929553564 time 777970485950 state 0x0 ref 1 00000100:00100000:3.0:1789760788.639570:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484362368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760788.639636:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25337856, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:3.0:1789760788.639647:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.639671:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=777s v=(0 0) 00010000:00010000:3.0:1789760788.639677:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760788.639685:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25337856->25341951] (req 25337856->25341951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.639700:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25337856->25341951] (req 25337856->25341951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760788.694675:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25337856 -> 25341952 00010000:00010000:3.0:1789760788.694705:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25337856->25341951] (req 25337856->25341951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.694722:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25337856->25341951] (req 25337856->25341951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.694738:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25337856->25341951] (req 25337856->25341951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.694759:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760788.694764:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25337856->25341951] (req 25337856->25341951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.694775:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25337856->25341951] (req 25337856->25341951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.694804:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25337856->25341951] (req 25337856->25341951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.694813:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2e3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25337856->25341951] (req 25337856->25341951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760788.694940:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484362368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55366us (55631us total) trans 4294973504 rc 0/0 00000100:00100000:3.0:1789760788.694953:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6215 00000100:00100000:3.0:1789760788.694976:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777929553564 time 777970485950 state 0x2 ref 0 00000001:00080000:0.0:1789760788.695987:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973504 is committed 00000100:00100000:0.0:1789760788.704323:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760788.705204:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484362496 00000100:00100000:1.0:1789760788.705219:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484362496 00000100:00100000:1.0:1789760788.705288:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777929553564 time 777970485950 state 0x0 ref 1 00000100:00100000:1.0:1789760788.705303:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777970746366 time 777970485950 state 0x1 ref 1 00000100:00100000:1.0:1789760788.705332:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777970746366 time 778036264180 state 0x0 ref 1 00000100:00100000:1.0:1789760788.705343:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484362496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760788.705404:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25341952, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:1.0:1789760788.705413:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.705429:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:1.0:1789760788.705433:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760788.705439:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25341952->25346047] (req 25341952->25346047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.705451:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25341952->25346047] (req 25341952->25346047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760788.710350:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760788.710397:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760788.710425:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760788.710443:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760788.710466:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760788.710502:0:6466:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484362624 00000100:00100000:0.0:1789760788.710514:0:6466:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484362624 00000100:00100000:0.0:1789760788.710594:0:6466:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 497 00000100:00100000:0.0:1789760788.710607:0:6466:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c803b850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484362624:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760788.710710:0:6466:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c803b850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:2627c619-a8ae-4613-a619-f7e733291af3+13:2423:x1876699484362624:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 106us (377us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760788.710718:0:6466:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 497 00000100:00100000:1.0:1789760788.710918:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484362880 00000100:00100000:1.0:1789760788.710922:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484362880 00000100:00100000:1.0:1789760788.710981:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484363136 00000100:00100000:1.0:1789760788.710984:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484363136 00000100:00100000:1.0:1789760788.711007:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484362752 00000100:00100000:1.0:1789760788.711011:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484362752 00000100:00100000:1.0:1789760788.711023:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484363008 00000100:00100000:1.0:1789760788.711025:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484363008 00000100:00100000:1.0:1789760788.711041:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1011 00000100:00100000:1.0:1789760788.711048:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98d8700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484362880:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760788.711235:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484362880:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 186us (837us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760788.711242:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1011 00000100:00100000:1.0:1789760788.711257:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1012 00000100:00100000:1.0:1789760788.711264:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484363136:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760788.711285:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484363136:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 22us (860us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760788.711291:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1012 00000100:00100000:1.0:1789760788.711300:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1013 00000100:00100000:1.0:1789760788.711306:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484362752:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760788.711325:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2423:x1876699484362752:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 19us (882us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760788.711331:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1013 00000100:00100000:1.0:1789760788.711339:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1014 00000100:00100000:1.0:1789760788.711346:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484363008:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760788.711365:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2423:x1876699484363008:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 19us (898us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760788.711371:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1014 00000100:00080000:0.0:1789760788.712743:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:0.0:1789760788.712754:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000001:00080000:3.0:1789760788.782555:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973505 is committed 00002000:00010000:1.0:1789760788.782689:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25341952 -> 25346048 00010000:00010000:1.0:1789760788.782709:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25341952->25346047] (req 25341952->25346047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.782724:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25341952->25346047] (req 25341952->25346047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.782734:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25341952->25346047] (req 25341952->25346047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.782742:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760788.782861:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25341952->25346047] (req 25341952->25346047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.782874:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25341952->25346047] (req 25341952->25346047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.782899:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25341952->25346047] (req 25341952->25346047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760788.782907:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a2ea lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25341952->25346047] (req 25341952->25346047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760788.783172:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98d8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484362496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77822us (78858us total) trans 4294973505 rc 0/0 00000100:00100000:1.0:1789760788.783186:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6216 00000100:00100000:1.0:1789760788.783199:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777970746366 time 778036264180 state 0x2 ref 0 00000100:00100000:0.0:1789760788.790783:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760788.790929:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484363264 00000100:00100000:2.0:1789760788.790943:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484363264 00000100:00100000:2.0:1789760788.791029:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 777970746366 time 778036264180 state 0x0 ref 1 00000100:00100000:2.0:1789760788.791049:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778036524596 time 778036264180 state 0x1 ref 1 00000100:00100000:2.0:1789760788.791128:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778036524596 time 778122057848 state 0x0 ref 1 00000100:00100000:2.0:1789760788.791143:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484363264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760788.791209:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25346048, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:2.0:1789760788.791221:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.791251:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:2.0:1789760788.791256:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760788.791443:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25346048->25350143] (req 25346048->25350143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.791468:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25346048->25350143] (req 25346048->25350143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760788.863126:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973506 is committed 00002000:00010000:2.0:1789760788.863387:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25346048 -> 25350144 00010000:00010000:2.0:1789760788.863409:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25346048->25350143] (req 25346048->25350143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.863424:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25346048->25350143] (req 25346048->25350143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.863435:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25346048->25350143] (req 25346048->25350143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.863444:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760788.863449:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25346048->25350143] (req 25346048->25350143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.863459:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25346048->25350143] (req 25346048->25350143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.863484:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25346048->25350143] (req 25346048->25350143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760788.863492:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a2f1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25346048->25350143] (req 25346048->25350143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760788.863689:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484363264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 72541us (72919us total) trans 4294973506 rc 0/0 00000100:00100000:2.0:1789760788.863879:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6217 00000100:00100000:2.0:1789760788.863892:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778036524596 time 778122057848 state 0x2 ref 0 00000100:00100000:2.0:1789760788.869653:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760788.869786:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484363392 00000100:00100000:3.0:1789760788.869800:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484363392 00000100:00100000:3.0:1789760788.869873:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778036524596 time 778122057848 state 0x0 ref 1 00000100:00100000:3.0:1789760788.869894:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778122318264 time 778122057848 state 0x1 ref 1 00000100:00100000:3.0:1789760788.869933:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778122318264 time 778200863253 state 0x0 ref 1 00000100:00100000:3.0:1789760788.869947:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484363392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760788.870025:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25350144, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:3.0:1789760788.870037:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.870104:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:3.0:1789760788.870110:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760788.870123:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25350144->25354239] (req 25350144->25354239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.870141:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25350144->25354239] (req 25350144->25354239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760788.953582:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973507 is committed 00002000:00010000:0.0:1789760788.958233:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25350144 -> 25354240 00010000:00010000:0.0:1789760788.958254:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25350144->25354239] (req 25350144->25354239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.958269:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25350144->25354239] (req 25350144->25354239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.958278:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25350144->25354239] (req 25350144->25354239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.958290:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760788.958294:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25350144->25354239] (req 25350144->25354239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.958303:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25350144->25354239] (req 25350144->25354239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.958323:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25350144->25354239] (req 25350144->25354239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760788.958331:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a2f8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25350144->25354239] (req 25350144->25354239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760788.958432:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484363392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 88483us (88791us total) trans 4294973507 rc 0/0 00000100:00100000:0.0:1789760788.958444:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6218 00000100:00100000:0.0:1789760788.958454:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778122318264 time 778200863253 state 0x2 ref 0 00000100:00100000:0.0:1789760788.965419:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760788.965500:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484363520 00000100:00100000:3.0:1789760788.965509:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484363520 00000100:00100000:3.0:1789760788.965560:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778122318264 time 778200863253 state 0x0 ref 1 00000100:00100000:3.0:1789760788.965575:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778201123669 time 778200863253 state 0x1 ref 1 00000100:00100000:3.0:1789760788.965600:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778201123669 time 778296531389 state 0x0 ref 1 00000100:00100000:3.0:1789760788.965617:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484363520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760788.965665:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25354240, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:3.0:1789760788.965676:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.965693:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:3.0:1789760788.965698:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760788.965704:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25354240->25358335] (req 25354240->25358335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760788.965719:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25354240->25358335] (req 25354240->25358335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760789.048340:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973508 is committed 00002000:00010000:0.0:1789760789.048470:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25354240 -> 25358336 00010000:00010000:0.0:1789760789.048490:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25354240->25358335] (req 25354240->25358335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.048502:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25354240->25358335] (req 25354240->25358335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.048516:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25354240->25358335] (req 25354240->25358335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.048523:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760789.048528:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25354240->25358335] (req 25354240->25358335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.048537:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25354240->25358335] (req 25354240->25358335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.048555:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25354240->25358335] (req 25354240->25358335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.048562:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a2ff lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25354240->25358335] (req 25354240->25358335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760789.048659:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484363520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 83040us (83245us total) trans 4294973508 rc 0/0 00000100:00100000:0.0:1789760789.048669:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6219 00000100:00100000:0.0:1789760789.048678:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778201123669 time 778296531389 state 0x2 ref 0 00000100:00100000:2.0:1789760789.069236:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760789.069351:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484363648 00000100:00100000:1.0:1789760789.069366:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484363648 00000100:00100000:0.0:1789760789.069551:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778201123669 time 778296531389 state 0x0 ref 1 00000100:00100000:0.0:1789760789.069573:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778296791805 time 778296531389 state 0x1 ref 1 00000100:00100000:0.0:1789760789.069672:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778296791805 time 778400601267 state 0x0 ref 1 00000100:00100000:0.0:1789760789.069698:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484363648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760789.069776:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25358336, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:0.0:1789760789.069788:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.069807:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:0.0:1789760789.069812:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760789.069819:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25358336->25362431] (req 25358336->25362431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.069836:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25358336->25362431] (req 25358336->25362431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760789.167941:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973509 is committed 00002000:00010000:1.0:1789760789.170330:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25358336 -> 25362432 00010000:00010000:1.0:1789760789.170358:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25358336->25362431] (req 25358336->25362431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.170373:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25358336->25362431] (req 25358336->25362431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.170384:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25358336->25362431] (req 25358336->25362431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.170391:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760789.170398:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25358336->25362431] (req 25358336->25362431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.170408:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25358336->25362431] (req 25358336->25362431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.170432:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25358336->25362431] (req 25358336->25362431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.170446:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a306 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25358336->25362431] (req 25358336->25362431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.170917:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484363648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 101225us (101687us total) trans 4294973509 rc 0/0 00000100:00100000:1.0:1789760789.170930:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6220 00000100:00100000:1.0:1789760789.170943:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778296791805 time 778400601267 state 0x2 ref 0 00000100:00100000:2.0:1789760789.189187:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760789.189290:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484363776 00000100:00100000:0.0:1789760789.189305:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484363776 00000100:00100000:0.0:1789760789.189387:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778296791805 time 778400601267 state 0x0 ref 1 00000100:00100000:0.0:1789760789.189407:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778400861683 time 778400601267 state 0x1 ref 1 00000100:00100000:0.0:1789760789.189443:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778400861683 time 778520373246 state 0x0 ref 1 00000100:00100000:0.0:1789760789.189458:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484363776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760789.189526:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25362432, mtime: 0x6aad9514, atime: 0x0, ctime: 0x6aad9514, blocks: 0xc807 00010000:00010000:0.0:1789760789.189545:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.189564:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:0.0:1789760789.189569:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760789.189577:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25362432->25366527] (req 25362432->25366527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.189594:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25362432->25366527] (req 25362432->25366527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760789.232637:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973510 is committed 00002000:00010000:2.0:1789760789.232819:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25362432 -> 25366528 00002000:00010000:2.0:1789760789.232824:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760788 -> 1789760789 00002000:00010000:2.0:1789760789.232828:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760788 -> 1789760789 00010000:00010000:2.0:1789760789.232846:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25362432->25366527] (req 25362432->25366527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.232862:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25362432->25366527] (req 25362432->25366527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.232872:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25362432->25366527] (req 25362432->25366527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.232881:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760789.232888:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25362432->25366527] (req 25362432->25366527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.232899:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25362432->25366527] (req 25362432->25366527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.232926:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25362432->25366527] (req 25362432->25366527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.232935:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a30d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25362432->25366527] (req 25362432->25366527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760789.233112:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484363776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43650us (43933us total) trans 4294973510 rc 0/0 00000100:00100000:2.0:1789760789.233127:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6221 00000100:00100000:2.0:1789760789.233139:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778400861683 time 778520373246 state 0x2 ref 0 00000100:00100000:1.0:1789760789.236827:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760789.236982:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484363904 00000100:00100000:3.0:1789760789.236995:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484363904 00000100:00100000:3.0:1789760789.237098:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778400861683 time 778520373246 state 0x0 ref 1 00000100:00100000:3.0:1789760789.237119:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778520633662 time 778520373246 state 0x1 ref 1 00000100:00100000:3.0:1789760789.237153:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778520633662 time 778568082979 state 0x0 ref 1 00000100:00100000:3.0:1789760789.237167:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484363904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760789.237241:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25366528, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:3.0:1789760789.237250:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.237272:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:3.0:1789760789.237277:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760789.237284:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25366528->25370623] (req 25366528->25370623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.237300:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25366528->25370623] (req 25366528->25370623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760789.289290:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25366528 -> 25370624 00010000:00010000:1.0:1789760789.289320:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25366528->25370623] (req 25366528->25370623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.289335:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25366528->25370623] (req 25366528->25370623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.289345:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25366528->25370623] (req 25366528->25370623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.289354:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760789.289360:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25366528->25370623] (req 25366528->25370623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.289370:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25366528->25370623] (req 25366528->25370623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.289396:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25366528->25370623] (req 25366528->25370623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.289405:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a314 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25366528->25370623] (req 25366528->25370623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.289547:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defc5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484363904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52368us (52720us total) trans 4294973511 rc 0/0 00000100:00100000:1.0:1789760789.289560:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6222 00000100:00100000:1.0:1789760789.289572:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778520633662 time 778568082979 state 0x2 ref 0 00000001:00080000:1.0:1789760789.289626:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973511 is committed 00000100:00100000:1.0:1789760789.295283:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760789.295512:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484364032 00000100:00100000:3.0:1789760789.295537:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484364032 00000100:00100000:3.0:1789760789.295615:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778520633662 time 778568082979 state 0x0 ref 1 00000100:00100000:3.0:1789760789.295635:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778568343395 time 778568082979 state 0x1 ref 1 00000100:00100000:3.0:1789760789.295670:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778568343395 time 778626601051 state 0x0 ref 1 00000100:00100000:3.0:1789760789.295685:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484364032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760789.295755:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25370624, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:3.0:1789760789.295767:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.295787:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:3.0:1789760789.295792:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760789.295801:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25370624->25374719] (req 25370624->25374719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.295825:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25370624->25374719] (req 25370624->25374719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760789.372411:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973512 is committed 00002000:00010000:3.0:1789760789.372614:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25370624 -> 25374720 00010000:00010000:3.0:1789760789.372633:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25370624->25374719] (req 25370624->25374719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.372648:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25370624->25374719] (req 25370624->25374719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.372658:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25370624->25374719] (req 25370624->25374719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.372666:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760789.372672:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25370624->25374719] (req 25370624->25374719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.372683:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25370624->25374719] (req 25370624->25374719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.372707:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25370624->25374719] (req 25370624->25374719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.372715:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a31b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25370624->25374719] (req 25370624->25374719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760789.373007:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f44000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484364032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 77316us (77732us total) trans 4294973512 rc 0/0 00000100:00100000:3.0:1789760789.373020:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6223 00000100:00100000:3.0:1789760789.373032:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778568343395 time 778626601051 state 0x2 ref 0 00000100:00100000:3.0:1789760789.377209:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760789.377483:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484364160 00000100:00100000:1.0:1789760789.377496:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484364160 00000100:00100000:1.0:1789760789.377574:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778568343395 time 778626601051 state 0x0 ref 1 00000100:00100000:1.0:1789760789.377594:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778626861467 time 778626601051 state 0x1 ref 1 00000100:00100000:1.0:1789760789.377630:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778626861467 time 778708560737 state 0x0 ref 1 00000100:00100000:1.0:1789760789.377643:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760789.377712:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25374720, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:1.0:1789760789.377722:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.377759:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:1.0:1789760789.377765:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760789.377774:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25374720->25378815] (req 25374720->25378815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.377792:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25374720->25378815] (req 25374720->25378815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.388538:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.388570:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c5f46300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508943104:0@lo:13:osp-pre-0-1.0 00000100:00100000:1.0:1789760789.388664:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760789.389420:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508943104 00000100:00100000:1.0:1789760789.389434:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508943104 00000100:00100000:1.0:1789760789.389510:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 418 00000100:00100000:1.0:1789760789.389521:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508943104:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:1.0:1789760789.389555:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528154624 pending 1703936 free 3832545280 avail 3800039424 00000100:00100000:1.0:1789760789.389622:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba76a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508943104:12345-0@lo:13:osp-pre-0-1.0 Request processed in 102us (957us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760789.389631:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 418 00000100:00100000:1.0:1789760789.389688:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c5f46300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508943104:0@lo:13:osp-pre-0-1.0 00000100:00080000:2.0:1789760789.390013:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00010000:00010000:1.0:1789760789.390164:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760789.390990:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fba74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508943232:0@lo:41:osp-pre-0-1.0 00000100:00100000:0.0:1789760789.391093:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760789.391244:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508943232 00000100:00100000:1.0:1789760789.391253:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508943232 00000100:00100000:1.0:1789760789.391306:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 477 00000100:00100000:1.0:1789760789.391316:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f58e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508943232:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:1.0:1789760789.391365:0:12534:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:1.0:1789760789.391412:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f58e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508943232:12345-0@lo:41:osp-pre-0-1.0 Request processed in 98us (319us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760789.391420:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 477 00000100:00100000:2.0:1789760789.391607:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:2.0:1789760789.391620:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4df82b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508943360:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1789760789.391659:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760789.391831:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508943360 00000100:00100000:0.0:1789760789.391836:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508943360 00000100:00100000:0.0:1789760789.391889:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 419 00000100:00100000:0.0:1789760789.391897:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508943360:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:0.0:1789760789.391912:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528154624 pending 1703936 free 3832545280 avail 3800039424 00000100:00100000:0.0:1789760789.391968:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508943360:12345-0@lo:13:osp-pre-0-0.0 Request processed in 63us (299us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760789.391975:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 419 00000100:00100000:0.0:1789760789.392015:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4df82b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508943360:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1789760789.392169:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fba74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508943232:0@lo:41:osp-pre-0-1.0 00000100:00080000:2.0:1789760789.392224:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:1.0:1789760789.424328:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25374720 -> 25378816 00010000:00010000:1.0:1789760789.424354:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25374720->25378815] (req 25374720->25378815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.424370:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25374720->25378815] (req 25374720->25378815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.424381:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25374720->25378815] (req 25374720->25378815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.424390:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760789.424396:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25374720->25378815] (req 25374720->25378815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.424406:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25374720->25378815] (req 25374720->25378815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.424434:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25374720->25378815] (req 25374720->25378815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.424443:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a322 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25374720->25378815] (req 25374720->25378815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.424572:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484364160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46924us (47372us total) trans 4294973513 rc 0/0 00000100:00100000:1.0:1789760789.424585:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6224 00000100:00100000:1.0:1789760789.424597:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778626861467 time 778708560737 state 0x2 ref 0 00000001:00080000:0.0:1789760789.424687:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973513 is committed 00000100:00100000:1.0:1789760789.427914:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760789.428194:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484364288 00000100:00100000:2.0:1789760789.428208:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484364288 00000100:00100000:2.0:1789760789.428281:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778626861467 time 778708560737 state 0x0 ref 1 00000100:00100000:2.0:1789760789.428301:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778708821153 time 778708560737 state 0x1 ref 1 00000100:00100000:2.0:1789760789.428373:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778708821153 time 778759302097 state 0x0 ref 1 00000100:00100000:2.0:1789760789.428389:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760789.428459:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25378816, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:2.0:1789760789.428471:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.428491:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:2.0:1789760789.428496:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760789.428504:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25378816->25382911] (req 25378816->25382911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.428521:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25378816->25382911] (req 25378816->25382911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760789.467299:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973514 is committed 00002000:00010000:1.0:1789760789.467445:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25378816 -> 25382912 00010000:00010000:1.0:1789760789.467467:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25378816->25382911] (req 25378816->25382911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.467485:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25378816->25382911] (req 25378816->25382911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.467494:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25378816->25382911] (req 25378816->25382911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.467503:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760789.467509:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25378816->25382911] (req 25378816->25382911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.467520:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25378816->25382911] (req 25378816->25382911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.467547:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25378816->25382911] (req 25378816->25382911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.467561:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a329 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25378816->25382911] (req 25378816->25382911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.467692:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39299us (39786us total) trans 4294973514 rc 0/0 00000100:00100000:1.0:1789760789.467706:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6225 00000100:00100000:1.0:1789760789.467720:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778708821153 time 778759302097 state 0x2 ref 0 00000100:00100000:3.0:1789760789.470774:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760789.471234:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484364416 00000100:00100000:0.0:1789760789.471248:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484364416 00000100:00100000:0.0:1789760789.471330:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778708821153 time 778759302097 state 0x0 ref 1 00000100:00100000:0.0:1789760789.471350:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778759562513 time 778759302097 state 0x1 ref 1 00000100:00100000:0.0:1789760789.471386:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778759562513 time 778802316354 state 0x0 ref 1 00000100:00100000:0.0:1789760789.471402:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760789.471469:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25382912, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:0.0:1789760789.471480:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.471500:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:0.0:1789760789.471505:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760789.471513:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25382912->25387007] (req 25382912->25387007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.471529:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25382912->25387007] (req 25382912->25387007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760789.506913:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973515 is committed 00002000:00010000:0.0:1789760789.507225:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25382912 -> 25387008 00010000:00010000:0.0:1789760789.507247:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25382912->25387007] (req 25382912->25387007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.507262:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25382912->25387007] (req 25382912->25387007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.507272:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25382912->25387007] (req 25382912->25387007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.507280:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760789.507286:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25382912->25387007] (req 25382912->25387007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.507297:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25382912->25387007] (req 25382912->25387007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.507324:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25382912->25387007] (req 25382912->25387007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.507334:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a330 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25382912->25387007] (req 25382912->25387007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760789.507522:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36116us (36758us total) trans 4294973515 rc 0/0 00000100:00100000:0.0:1789760789.507535:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6226 00000100:00100000:0.0:1789760789.507547:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778759562513 time 778802316354 state 0x2 ref 0 00000100:00100000:3.0:1789760789.511660:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760789.511844:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484364544 00000100:00100000:0.0:1789760789.511858:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484364544 00000100:00100000:0.0:1789760789.511940:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778759562513 time 778802316354 state 0x0 ref 1 00000100:00100000:0.0:1789760789.511971:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778802576770 time 778802316354 state 0x1 ref 1 00000100:00100000:0.0:1789760789.512056:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778802576770 time 778842979841 state 0x0 ref 1 00000100:00100000:0.0:1789760789.512098:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760789.512166:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25387008, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:0.0:1789760789.512176:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.512195:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:0.0:1789760789.512201:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760789.512209:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25387008->25391103] (req 25387008->25391103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.512225:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25387008->25391103] (req 25387008->25391103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760789.550181:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973516 is committed 00002000:00010000:0.0:1789760789.550315:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25387008 -> 25391104 00010000:00010000:0.0:1789760789.550334:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25387008->25391103] (req 25387008->25391103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.550350:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25387008->25391103] (req 25387008->25391103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.550361:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25387008->25391103] (req 25387008->25391103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.550369:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760789.550376:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25387008->25391103] (req 25387008->25391103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.550387:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25387008->25391103] (req 25387008->25391103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.550413:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25387008->25391103] (req 25387008->25391103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.550422:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a337 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25387008->25391103] (req 25387008->25391103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760789.550546:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38447us (38893us total) trans 4294973516 rc 0/0 00000100:00100000:0.0:1789760789.550558:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6227 00000100:00100000:0.0:1789760789.550571:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778802576770 time 778842979841 state 0x2 ref 0 00000100:00100000:1.0:1789760789.553518:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760789.554022:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484364672 00000100:00100000:0.0:1789760789.554035:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484364672 00000100:00100000:1.0:1789760789.554240:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778802576770 time 778842979841 state 0x0 ref 1 00000100:00100000:1.0:1789760789.554259:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778843240257 time 778842979841 state 0x1 ref 1 00000100:00100000:1.0:1789760789.554293:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778843240257 time 778885223873 state 0x0 ref 1 00000100:00100000:1.0:1789760789.554309:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba75180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760789.554378:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25391104, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:1.0:1789760789.554389:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.554409:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:1.0:1789760789.554414:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760789.554422:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25391104->25395199] (req 25391104->25395199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.554445:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25391104->25395199] (req 25391104->25395199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760789.591670:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25391104 -> 25395200 00010000:00010000:2.0:1789760789.591695:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25391104->25395199] (req 25391104->25395199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.591718:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25391104->25395199] (req 25391104->25395199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.591728:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25391104->25395199] (req 25391104->25395199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.591747:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760789.591754:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25391104->25395199] (req 25391104->25395199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.591765:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25391104->25395199] (req 25391104->25395199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.591791:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25391104->25395199] (req 25391104->25395199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.591799:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a33e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25391104->25395199] (req 25391104->25395199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760789.591936:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba75180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484364672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37623us (38427us total) trans 4294973517 rc 0/0 00000100:00100000:2.0:1789760789.591950:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6228 00000100:00100000:2.0:1789760789.591973:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778843240257 time 778885223873 state 0x2 ref 0 00000001:00080000:3.0:1789760789.592763:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973517 is committed 00000100:00100000:1.0:1789760789.595916:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760789.596016:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484364800 00000100:00100000:2.0:1789760789.596030:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484364800 00000100:00100000:2.0:1789760789.596163:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778843240257 time 778885223873 state 0x0 ref 1 00000100:00100000:2.0:1789760789.596182:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778885484289 time 778885223873 state 0x1 ref 1 00000100:00100000:2.0:1789760789.596210:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778885484289 time 778927140546 state 0x0 ref 1 00000100:00100000:2.0:1789760789.596224:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760789.596289:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25395200, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:2.0:1789760789.596299:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.596318:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:2.0:1789760789.596323:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760789.596330:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25395200->25399295] (req 25395200->25399295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.596345:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25395200->25399295] (req 25395200->25399295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760789.624509:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973518 is committed 00002000:00010000:2.0:1789760789.624629:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25395200 -> 25399296 00010000:00010000:2.0:1789760789.624652:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25395200->25399295] (req 25395200->25399295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.624670:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25395200->25399295] (req 25395200->25399295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.624686:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25395200->25399295] (req 25395200->25399295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.624694:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760789.624699:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25395200->25399295] (req 25395200->25399295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.624710:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25395200->25399295] (req 25395200->25399295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.624736:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25395200->25399295] (req 25395200->25399295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.624824:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a345 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25395200->25399295] (req 25395200->25399295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760789.625016:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28787us (29118us total) trans 4294973518 rc 0/0 00000100:00100000:2.0:1789760789.625029:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6229 00000100:00100000:2.0:1789760789.625041:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778885484289 time 778927140546 state 0x2 ref 0 00000100:00100000:1.0:1789760789.629282:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760789.629399:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484364928 00000100:00100000:0.0:1789760789.629411:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484364928 00000100:00100000:0.0:1789760789.629493:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778885484289 time 778927140546 state 0x0 ref 1 00000100:00100000:0.0:1789760789.629512:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778927400962 time 778927140546 state 0x1 ref 1 00000100:00100000:0.0:1789760789.629570:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778927400962 time 778960500682 state 0x0 ref 1 00000100:00100000:0.0:1789760789.629588:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760789.629655:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25399296, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:0.0:1789760789.629666:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.629685:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:0.0:1789760789.629690:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760789.629697:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25399296->25403391] (req 25399296->25403391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.629712:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25399296->25403391] (req 25399296->25403391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760789.661801:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973519 is committed 00002000:00010000:1.0:1789760789.662001:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25399296 -> 25403392 00010000:00010000:1.0:1789760789.662022:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25399296->25403391] (req 25399296->25403391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.662036:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25399296->25403391] (req 25399296->25403391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.662057:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25399296->25403391] (req 25399296->25403391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.662094:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760789.662100:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25399296->25403391] (req 25399296->25403391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.662110:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25399296->25403391] (req 25399296->25403391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.662136:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25399296->25403391] (req 25399296->25403391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.662151:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000622517e1/0x2cf382639620a34c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25399296->25403391] (req 25399296->25403391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.662266:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484364928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32676us (32992us total) trans 4294973519 rc 0/0 00000100:00100000:1.0:1789760789.662277:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6230 00000100:00100000:1.0:1789760789.662289:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778927400962 time 778960500682 state 0x2 ref 0 00000100:00100000:1.0:1789760789.665001:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760789.665307:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484365056 00000100:00100000:0.0:1789760789.665319:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484365056 00000100:00100000:0.0:1789760789.665397:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778927400962 time 778960500682 state 0x0 ref 1 00000100:00100000:0.0:1789760789.665415:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778960761098 time 778960500682 state 0x1 ref 1 00000100:00100000:0.0:1789760789.665486:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778960761098 time 778996415824 state 0x0 ref 1 00000100:00100000:0.0:1789760789.665502:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484365056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760789.665565:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25403392, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:0.0:1789760789.665576:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.665598:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=778s v=(0 0) 00010000:00010000:0.0:1789760789.665604:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760789.665611:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25403392->25407487] (req 25403392->25407487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.665628:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25403392->25407487] (req 25403392->25407487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760789.694773:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973520 is committed 00002000:00010000:1.0:1789760789.694980:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25403392 -> 25407488 00010000:00010000:1.0:1789760789.695002:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25403392->25407487] (req 25403392->25407487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.695017:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25403392->25407487] (req 25403392->25407487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.695027:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25403392->25407487] (req 25403392->25407487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.695036:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760789.695128:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25403392->25407487] (req 25403392->25407487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.695142:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25403392->25407487] (req 25403392->25407487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.695176:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25403392->25407487] (req 25403392->25407487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.695185:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a353 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25403392->25407487] (req 25403392->25407487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.695307:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba75c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484365056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29802us (30312us total) trans 4294973520 rc 0/0 00000100:00100000:1.0:1789760789.695319:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6231 00000100:00100000:1.0:1789760789.695332:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778960761098 time 778996415824 state 0x2 ref 0 00000100:00100000:1.0:1789760789.699965:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760789.700165:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484365184 00000100:00100000:2.0:1789760789.700179:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484365184 00000100:00100000:2.0:1789760789.700253:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778960761098 time 778996415824 state 0x0 ref 1 00000100:00100000:2.0:1789760789.700275:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778996676240 time 778996415824 state 0x1 ref 1 00000100:00100000:2.0:1789760789.700346:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778996676240 time 779031276605 state 0x0 ref 1 00000100:00100000:2.0:1789760789.700361:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484365184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760789.700426:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25407488, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:2.0:1789760789.700437:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.700456:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:2.0:1789760789.700461:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760789.700469:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25407488->25411583] (req 25407488->25411583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.700486:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25407488->25411583] (req 25407488->25411583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760789.733614:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25407488 -> 25411584 00010000:00010000:2.0:1789760789.733642:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25407488->25411583] (req 25407488->25411583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.733658:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25407488->25411583] (req 25407488->25411583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.733668:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25407488->25411583] (req 25407488->25411583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.733682:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760789.733688:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25407488->25411583] (req 25407488->25411583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.733698:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25407488->25411583] (req 25407488->25411583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.733724:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25407488->25411583] (req 25407488->25411583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.733735:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a35a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25407488->25411583] (req 25407488->25411583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760789.733941:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fba74380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484365184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33575us (33994us total) trans 4294973521 rc 0/0 00000100:00100000:2.0:1789760789.733990:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6232 00000100:00100000:2.0:1789760789.734005:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778996676240 time 779031276605 state 0x2 ref 0 00000001:00080000:3.0:1789760789.736939:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973521 is committed 00000100:00100000:2.0:1789760789.737252:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760789.737408:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484365312 00000100:00100000:3.0:1789760789.737420:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484365312 00000100:00100000:3.0:1789760789.737492:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 778996676240 time 779031276605 state 0x0 ref 1 00000100:00100000:3.0:1789760789.737512:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779031537021 time 779031276605 state 0x1 ref 1 00000100:00100000:3.0:1789760789.737548:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779031537021 time 779068478942 state 0x0 ref 1 00000100:00100000:3.0:1789760789.737563:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484365312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760789.737630:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25411584, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:3.0:1789760789.737641:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.737663:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:3.0:1789760789.737668:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760789.737676:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25411584->25415679] (req 25411584->25415679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.737692:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25411584->25415679] (req 25411584->25415679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760789.778158:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973522 is committed 00002000:00010000:2.0:1789760789.778356:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25411584 -> 25415680 00010000:00010000:2.0:1789760789.778377:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25411584->25415679] (req 25411584->25415679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.778393:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25411584->25415679] (req 25411584->25415679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.778410:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25411584->25415679] (req 25411584->25415679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.778419:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760789.778425:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25411584->25415679] (req 25411584->25415679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.778435:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25411584->25415679] (req 25411584->25415679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.778462:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25411584->25415679] (req 25411584->25415679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.778471:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a361 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25411584->25415679] (req 25411584->25415679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760789.778611:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484365312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41043us (41357us total) trans 4294973522 rc 0/0 00000100:00100000:2.0:1789760789.778624:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6233 00000100:00100000:2.0:1789760789.778637:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779031537021 time 779068478942 state 0x2 ref 0 00000100:00100000:2.0:1789760789.782647:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760789.782810:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484365440 00000100:00100000:1.0:1789760789.782824:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484365440 00000100:00100000:1.0:1789760789.782896:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779031537021 time 779068478942 state 0x0 ref 1 00000100:00100000:1.0:1789760789.782915:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779068739358 time 779068478942 state 0x1 ref 1 00000100:00100000:1.0:1789760789.783059:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779068739358 time 779113988542 state 0x0 ref 1 00000100:00100000:1.0:1789760789.783114:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484365440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760789.783184:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25415680, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:1.0:1789760789.783196:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.783222:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:1.0:1789760789.783228:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760789.783236:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25415680->25419775] (req 25415680->25419775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.783253:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25415680->25419775] (req 25415680->25419775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760789.828268:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25415680 -> 25419776 00010000:00010000:1.0:1789760789.828299:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25415680->25419775] (req 25415680->25419775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.828316:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25415680->25419775] (req 25415680->25419775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.828327:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25415680->25419775] (req 25415680->25419775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.828337:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760789.828342:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25415680->25419775] (req 25415680->25419775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.828352:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25415680->25419775] (req 25415680->25419775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.828380:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25415680->25419775] (req 25415680->25419775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.828389:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a368 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25415680->25419775] (req 25415680->25419775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.828516:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484365440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45398us (45875us total) trans 4294973523 rc 0/0 00000100:00100000:1.0:1789760789.828529:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6234 00000100:00100000:1.0:1789760789.828541:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779068739358 time 779113988542 state 0x2 ref 0 00000001:00080000:3.0:1789760789.829660:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973523 is committed 00000100:00100000:1.0:1789760789.831447:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760789.831552:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484365568 00000100:00100000:3.0:1789760789.831566:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484365568 00000100:00100000:3.0:1789760789.831650:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779068739358 time 779113988542 state 0x0 ref 1 00000100:00100000:3.0:1789760789.831670:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779114248958 time 779113988542 state 0x1 ref 1 00000100:00100000:3.0:1789760789.831698:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779114248958 time 779162629359 state 0x0 ref 1 00000100:00100000:3.0:1789760789.831713:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c5f46300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484365568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760789.831794:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25419776, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:3.0:1789760789.831805:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.831829:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:3.0:1789760789.831834:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760789.831843:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25419776->25423871] (req 25419776->25423871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.831861:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25419776->25423871] (req 25419776->25423871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760789.872026:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973524 is committed 00002000:00010000:3.0:1789760789.872417:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25419776 -> 25423872 00010000:00010000:3.0:1789760789.872438:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25419776->25423871] (req 25419776->25423871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.872466:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25419776->25423871] (req 25419776->25423871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.872478:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25419776->25423871] (req 25419776->25423871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.872486:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760789.872491:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25419776->25423871] (req 25419776->25423871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.872502:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25419776->25423871] (req 25419776->25423871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.872528:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25419776->25423871] (req 25419776->25423871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760789.872536:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a36f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25419776->25423871] (req 25419776->25423871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760789.872676:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c5f46300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484365568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40958us (41236us total) trans 4294973524 rc 0/0 00000100:00100000:3.0:1789760789.872689:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6235 00000100:00100000:3.0:1789760789.872702:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779114248958 time 779162629359 state 0x2 ref 0 00000100:00100000:1.0:1789760789.880610:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760789.880728:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484365696 00000100:00100000:2.0:1789760789.880741:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484365696 00000100:00100000:2.0:1789760789.880816:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779114248958 time 779162629359 state 0x0 ref 1 00000100:00100000:2.0:1789760789.880839:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779162889775 time 779162629359 state 0x1 ref 1 00000100:00100000:2.0:1789760789.880874:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779162889775 time 779211805269 state 0x0 ref 1 00000100:00100000:2.0:1789760789.880889:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484365696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760789.880965:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25423872, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:2.0:1789760789.880975:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.880995:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:2.0:1789760789.881000:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760789.881008:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25423872->25427967] (req 25423872->25427967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760789.881026:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25423872->25427967] (req 25423872->25427967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.902360:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c98dea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508943488:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760789.902458:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760789.902566:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508943488 00000100:00100000:1.0:1789760789.902577:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508943488 00000100:00100000:1.0:1789760789.902639:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 478 00000100:00100000:1.0:1789760789.902650:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508943488:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760789.902706:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760789.902759:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508943488:12345-0@lo:41:osp-pre-1-0.0 Request processed in 111us (301us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760789.902766:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 478 00000100:00100000:1.0:1789760789.902838:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c98dea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508943488:0@lo:41:osp-pre-1-0.0 00000001:00080000:3.0:1789760789.989609:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973525 is committed 00002000:00010000:1.0:1789760789.989735:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25423872 -> 25427968 00010000:00010000:1.0:1789760789.989759:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25423872->25427967] (req 25423872->25427967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.989779:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25423872->25427967] (req 25423872->25427967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.989789:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25423872->25427967] (req 25423872->25427967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.989796:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760789.989802:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25423872->25427967] (req 25423872->25427967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.989811:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25423872->25427967] (req 25423872->25427967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.989837:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25423872->25427967] (req 25423872->25427967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760789.989845:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a376 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25423872->25427967] (req 25423872->25427967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760789.989983:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484365696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 109090us (109380us total) trans 4294973525 rc 0/0 00000100:00100000:1.0:1789760789.989996:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6236 00000100:00100000:1.0:1789760789.990008:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779162889775 time 779211805269 state 0x2 ref 0 00000100:00100000:2.0:1789760789.994395:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760789.994488:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484365824 00000100:00100000:0.0:1789760789.994502:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484365824 00000100:00100000:0.0:1789760789.994590:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779162889775 time 779211805269 state 0x0 ref 1 00000100:00100000:0.0:1789760789.994610:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779212065685 time 779211805269 state 0x1 ref 1 00000100:00100000:0.0:1789760789.994649:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779212065685 time 779325580032 state 0x0 ref 1 00000100:00100000:0.0:1789760789.994663:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484365824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760789.994731:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25427968, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:0.0:1789760789.994741:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.994761:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:0.0:1789760789.994766:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760789.994774:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25427968->25432063] (req 25427968->25432063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760789.994792:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25427968->25432063] (req 25427968->25432063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760790.049448:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25427968 -> 25432064 00010000:00010000:3.0:1789760790.049477:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25427968->25432063] (req 25427968->25432063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.049494:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25427968->25432063] (req 25427968->25432063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.049503:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25427968->25432063] (req 25427968->25432063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.049511:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760790.049516:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25427968->25432063] (req 25427968->25432063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.049525:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25427968->25432063] (req 25427968->25432063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.049550:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25427968->25432063] (req 25427968->25432063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.049565:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a37d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25427968->25432063] (req 25427968->25432063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760790.049696:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484365824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55028us (55310us total) trans 4294973526 rc 0/0 00000100:00100000:3.0:1789760790.049709:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6237 00000100:00100000:3.0:1789760790.049721:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779212065685 time 779325580032 state 0x2 ref 0 00000001:00080000:3.0:1789760790.049771:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973526 is committed 00000100:00100000:2.0:1789760790.056777:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760790.056970:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484365952 00000100:00100000:1.0:1789760790.056986:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484365952 00000100:00100000:1.0:1789760790.057141:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779212065685 time 779325580032 state 0x0 ref 1 00000100:00100000:1.0:1789760790.057163:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779325840448 time 779325580032 state 0x1 ref 1 00000100:00100000:1.0:1789760790.057372:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779325840448 time 779388300109 state 0x0 ref 1 00000100:00100000:1.0:1789760790.057391:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484365952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760790.057461:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25432064, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:1.0:1789760790.057472:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.057493:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:1.0:1789760790.057499:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760790.057507:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25432064->25436159] (req 25432064->25436159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.057525:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25432064->25436159] (req 25432064->25436159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760790.058139:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:3.0:1789760790.105753:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973527 is committed 00002000:00010000:1.0:1789760790.105921:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25432064 -> 25436160 00010000:00010000:1.0:1789760790.105941:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25432064->25436159] (req 25432064->25436159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.105973:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25432064->25436159] (req 25432064->25436159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.105989:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25432064->25436159] (req 25432064->25436159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.105997:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760790.106003:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25432064->25436159] (req 25432064->25436159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.106012:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25432064->25436159] (req 25432064->25436159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.106288:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25432064->25436159] (req 25432064->25436159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.106299:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a384 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25432064->25436159] (req 25432064->25436159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760790.106466:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484365952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49074us (49698us total) trans 4294973527 rc 0/0 00000100:00100000:1.0:1789760790.106477:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6238 00000100:00100000:1.0:1789760790.106490:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779325840448 time 779388300109 state 0x2 ref 0 00000100:00100000:2.0:1789760790.112195:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.112358:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484366080 00000100:00100000:3.0:1789760790.112377:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484366080 00000100:00100000:3.0:1789760790.112459:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779325840448 time 779388300109 state 0x0 ref 1 00000100:00100000:3.0:1789760790.112478:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779388560525 time 779388300109 state 0x1 ref 1 00000100:00100000:3.0:1789760790.112516:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779388560525 time 779443447076 state 0x0 ref 1 00000100:00100000:3.0:1789760790.112530:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484366080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.112598:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25436160, mtime: 0x6aad9515, atime: 0x0, ctime: 0x6aad9515, blocks: 0xc807 00010000:00010000:3.0:1789760790.112609:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.112631:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:3.0:1789760790.112636:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.112644:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25436160->25440255] (req 25436160->25440255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.112661:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25436160->25440255] (req 25436160->25440255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760790.157639:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973528 is committed 00002000:00010000:0.0:1789760790.157726:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25436160 -> 25440256 00002000:00010000:0.0:1789760790.157731:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760789 -> 1789760790 00002000:00010000:0.0:1789760790.157741:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760789 -> 1789760790 00010000:00010000:0.0:1789760790.157754:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25436160->25440255] (req 25436160->25440255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.157767:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25436160->25440255] (req 25436160->25440255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.157776:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25436160->25440255] (req 25436160->25440255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.157783:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760790.157789:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25436160->25440255] (req 25436160->25440255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.157797:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25436160->25440255] (req 25436160->25440255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.157815:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25436160->25440255] (req 25436160->25440255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.157822:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a38b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25436160->25440255] (req 25436160->25440255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760790.157926:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484366080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45395us (45735us total) trans 4294973528 rc 0/0 00000100:00100000:0.0:1789760790.157936:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6239 00000100:00100000:0.0:1789760790.157946:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779388560525 time 779443447076 state 0x2 ref 0 00000100:00100000:2.0:1789760790.161543:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.161634:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484366208 00000100:00100000:3.0:1789760790.161643:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484366208 00000100:00100000:3.0:1789760790.161695:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779388560525 time 779443447076 state 0x0 ref 1 00000100:00100000:3.0:1789760790.161710:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779443707492 time 779443447076 state 0x1 ref 1 00000100:00100000:3.0:1789760790.161740:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779443707492 time 779492671489 state 0x0 ref 1 00000100:00100000:3.0:1789760790.161752:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484366208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.161799:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25440256, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760790.161808:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.161823:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:3.0:1789760790.161827:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.161834:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25440256->25444351] (req 25440256->25444351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.161943:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25440256->25444351] (req 25440256->25444351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760790.200131:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25440256 -> 25444352 00010000:00010000:3.0:1789760790.200151:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25440256->25444351] (req 25440256->25444351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.200164:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25440256->25444351] (req 25440256->25444351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.200173:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25440256->25444351] (req 25440256->25444351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.200180:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760790.200185:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25440256->25444351] (req 25440256->25444351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.200194:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25440256->25444351] (req 25440256->25444351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.200213:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25440256->25444351] (req 25440256->25444351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.200220:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a392 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25440256->25444351] (req 25440256->25444351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760790.200299:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484366208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38544us (38763us total) trans 4294973529 rc 0/0 00000100:00100000:3.0:1789760790.200312:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6240 00000100:00100000:3.0:1789760790.200322:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779443707492 time 779492671489 state 0x2 ref 0 00000001:00080000:0.0:1789760790.200401:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973529 is committed 00000100:00100000:3.0:1789760790.203389:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760790.203672:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484366336 00000100:00100000:0.0:1789760790.203783:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484366336 00000100:00100000:0.0:1789760790.203851:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779443707492 time 779492671489 state 0x0 ref 1 00000100:00100000:0.0:1789760790.203866:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779492931905 time 779492671489 state 0x1 ref 1 00000100:00100000:0.0:1789760790.203928:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779492931905 time 779534858214 state 0x0 ref 1 00000100:00100000:0.0:1789760790.203942:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484366336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760790.204011:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25444352, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:0.0:1789760790.204019:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.204034:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:0.0:1789760790.204039:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760790.204045:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25444352->25448447] (req 25444352->25448447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.204060:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25444352->25448447] (req 25444352->25448447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760790.238858:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973530 is committed 00002000:00010000:3.0:1789760790.238979:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25444352 -> 25448448 00010000:00010000:3.0:1789760790.239134:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25444352->25448447] (req 25444352->25448447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.239148:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25444352->25448447] (req 25444352->25448447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.239158:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25444352->25448447] (req 25444352->25448447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.239165:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760790.239170:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25444352->25448447] (req 25444352->25448447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.239178:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25444352->25448447] (req 25444352->25448447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.239197:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25444352->25448447] (req 25444352->25448447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.239204:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a399 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25444352->25448447] (req 25444352->25448447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760790.239296:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484366336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35352us (35911us total) trans 4294973530 rc 0/0 00000100:00100000:3.0:1789760790.239305:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6241 00000100:00100000:3.0:1789760790.239314:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779492931905 time 779534858214 state 0x2 ref 0 00000100:00100000:1.0:1789760790.244594:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760790.244687:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484366464 00000100:00100000:2.0:1789760790.244697:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484366464 00000100:00100000:2.0:1789760790.244756:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779492931905 time 779534858214 state 0x0 ref 1 00000100:00100000:2.0:1789760790.244770:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779535118630 time 779534858214 state 0x1 ref 1 00000100:00100000:2.0:1789760790.244796:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779535118630 time 779575727531 state 0x0 ref 1 00000100:00100000:2.0:1789760790.244808:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484366464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760790.244853:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25448448, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:2.0:1789760790.244861:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.244874:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:2.0:1789760790.244878:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760790.244884:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25448448->25452543] (req 25448448->25452543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.244898:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25448448->25452543] (req 25448448->25452543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760790.303638:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973531 is committed 00002000:00010000:3.0:1789760790.303784:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25448448 -> 25452544 00010000:00010000:3.0:1789760790.303799:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25448448->25452543] (req 25448448->25452543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.303812:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25448448->25452543] (req 25448448->25452543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.303822:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25448448->25452543] (req 25448448->25452543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.303829:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760790.303833:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25448448->25452543] (req 25448448->25452543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.303890:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25448448->25452543] (req 25448448->25452543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.303911:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25448448->25452543] (req 25448448->25452543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.303925:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a3a0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25448448->25452543] (req 25448448->25452543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760790.304249:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484366464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59436us (59658us total) trans 4294973531 rc 0/0 00000100:00100000:3.0:1789760790.304260:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6242 00000100:00100000:3.0:1789760790.304280:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779535118630 time 779575727531 state 0x2 ref 0 00000100:00100000:0.0:1789760790.313947:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.314042:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484366592 00000100:00100000:3.0:1789760790.314052:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484366592 00000100:00100000:3.0:1789760790.314130:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779535118630 time 779575727531 state 0x0 ref 1 00000100:00100000:3.0:1789760790.314144:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779575987947 time 779575727531 state 0x1 ref 1 00000100:00100000:3.0:1789760790.314164:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779575987947 time 779645096145 state 0x0 ref 1 00000100:00100000:3.0:1789760790.314175:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484366592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.314223:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25452544, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760790.314232:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.314247:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:3.0:1789760790.314251:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.314257:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25452544->25456639] (req 25452544->25456639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.314276:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25452544->25456639] (req 25452544->25456639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760790.357092:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973532 is committed 00002000:00010000:3.0:1789760790.357523:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25452544 -> 25456640 00010000:00010000:3.0:1789760790.357540:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25452544->25456639] (req 25452544->25456639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.357554:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25452544->25456639] (req 25452544->25456639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.357563:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25452544->25456639] (req 25452544->25456639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.357570:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760790.357575:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25452544->25456639] (req 25452544->25456639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.357584:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25452544->25456639] (req 25452544->25456639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.357604:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25452544->25456639] (req 25452544->25456639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.357612:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3a7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25452544->25456639] (req 25452544->25456639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760790.357708:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484366592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43529us (43764us total) trans 4294973532 rc 0/0 00000100:00100000:3.0:1789760790.357718:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6243 00000100:00100000:3.0:1789760790.357728:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779575987947 time 779645096145 state 0x2 ref 0 00000100:00100000:3.0:1789760790.361026:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760790.361682:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484366720 00000100:00100000:1.0:1789760790.361694:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484366720 00000100:00100000:1.0:1789760790.361752:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779575987947 time 779645096145 state 0x0 ref 1 00000100:00100000:1.0:1789760790.361766:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779645356561 time 779645096145 state 0x1 ref 1 00000100:00100000:1.0:1789760790.361796:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779645356561 time 779692727000 state 0x0 ref 1 00000100:00100000:1.0:1789760790.361809:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484366720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760790.363149:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25456640, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:1.0:1789760790.363163:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.363180:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:1.0:1789760790.363184:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760790.363191:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25456640->25460735] (req 25456640->25460735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.363205:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25456640->25460735] (req 25456640->25460735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760790.391851:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973533 is committed 00002000:00010000:1.0:1789760790.392127:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25456640 -> 25460736 00010000:00010000:1.0:1789760790.392145:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25456640->25460735] (req 25456640->25460735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.392157:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25456640->25460735] (req 25456640->25460735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.392166:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25456640->25460735] (req 25456640->25460735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.392174:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760790.392179:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25456640->25460735] (req 25456640->25460735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.392187:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25456640->25460735] (req 25456640->25460735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.392207:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25456640->25460735] (req 25456640->25460735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.392215:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a3ae lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25456640->25460735] (req 25456640->25460735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760790.392338:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484366720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30527us (31317us total) trans 4294973533 rc 0/0 00000100:00100000:1.0:1789760790.392349:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6244 00000100:00100000:1.0:1789760790.392360:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779645356561 time 779692727000 state 0x2 ref 0 00000100:00100000:1.0:1789760790.395826:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760790.395893:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484366848 00000100:00100000:0.0:1789760790.395903:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484366848 00000100:00100000:0.0:1789760790.395978:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779645356561 time 779692727000 state 0x0 ref 1 00000100:00100000:0.0:1789760790.395993:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779692987416 time 779692727000 state 0x1 ref 1 00000100:00100000:0.0:1789760790.396022:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779692987416 time 779726952401 state 0x0 ref 1 00000100:00100000:0.0:1789760790.396036:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484366848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760790.396120:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25460736, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:0.0:1789760790.396130:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.396146:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:0.0:1789760790.396151:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760790.396158:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25460736->25464831] (req 25460736->25464831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.396170:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25460736->25464831] (req 25460736->25464831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.416125:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:1.0:1789760790.466859:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973534 is committed 00002000:00010000:0.0:1789760790.468032:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25460736 -> 25464832 00010000:00010000:0.0:1789760790.468058:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25460736->25464831] (req 25460736->25464831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.468108:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25460736->25464831] (req 25460736->25464831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.468126:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25460736->25464831] (req 25460736->25464831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.468134:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760790.468139:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25460736->25464831] (req 25460736->25464831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.468149:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25460736->25464831] (req 25460736->25464831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.468167:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25460736->25464831] (req 25460736->25464831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.468175:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3b5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25460736->25464831] (req 25460736->25464831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760790.468289:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484366848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 72252us (72468us total) trans 4294973534 rc 0/0 00000100:00100000:0.0:1789760790.468300:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6245 00000100:00100000:0.0:1789760790.468310:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779692987416 time 779726952401 state 0x2 ref 0 00000100:00100000:0.0:1789760790.476389:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.476472:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484366976 00000100:00100000:3.0:1789760790.476482:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484366976 00000100:00100000:3.0:1789760790.476538:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779692987416 time 779726952401 state 0x0 ref 1 00000100:00100000:3.0:1789760790.476553:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779727212817 time 779726952401 state 0x1 ref 1 00000100:00100000:3.0:1789760790.476585:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779727212817 time 779807516280 state 0x0 ref 1 00000100:00100000:3.0:1789760790.476599:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484366976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.476648:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25464832, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760790.476656:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.476673:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:3.0:1789760790.476677:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.476683:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25464832->25468927] (req 25464832->25468927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.476704:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25464832->25468927] (req 25464832->25468927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760790.526102:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973535 is committed 00002000:00010000:3.0:1789760790.526265:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25464832 -> 25468928 00010000:00010000:3.0:1789760790.526280:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25464832->25468927] (req 25464832->25468927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.526291:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25464832->25468927] (req 25464832->25468927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.526300:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25464832->25468927] (req 25464832->25468927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.526306:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760790.526311:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25464832->25468927] (req 25464832->25468927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.526320:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25464832->25468927] (req 25464832->25468927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.526338:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25464832->25468927] (req 25464832->25468927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.526345:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3bc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25464832->25468927] (req 25464832->25468927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760790.526622:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484366976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50020us (50238us total) trans 4294973535 rc 0/0 00000100:00100000:3.0:1789760790.526634:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6246 00000100:00100000:3.0:1789760790.526643:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779727212817 time 779807516280 state 0x2 ref 0 00000100:00100000:1.0:1789760790.531900:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.531981:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484367104 00000100:00100000:3.0:1789760790.531991:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484367104 00000100:00100000:3.0:1789760790.532042:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779727212817 time 779807516280 state 0x0 ref 1 00000100:00100000:3.0:1789760790.532057:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779807776696 time 779807516280 state 0x1 ref 1 00000100:00100000:3.0:1789760790.532121:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779807776696 time 779863051716 state 0x0 ref 1 00000100:00100000:3.0:1789760790.532134:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484367104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.532180:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25468928, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760790.532190:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.532204:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:3.0:1789760790.532208:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.532214:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25468928->25473023] (req 25468928->25473023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.532227:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25468928->25473023] (req 25468928->25473023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760790.595891:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973536 is committed 00002000:00010000:1.0:1789760790.595991:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25468928 -> 25473024 00010000:00010000:1.0:1789760790.596006:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25468928->25473023] (req 25468928->25473023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.596025:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25468928->25473023] (req 25468928->25473023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.596035:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25468928->25473023] (req 25468928->25473023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.596042:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760790.596047:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25468928->25473023] (req 25468928->25473023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.596056:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25468928->25473023] (req 25468928->25473023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.596102:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25468928->25473023] (req 25468928->25473023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.596111:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a3c3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25468928->25473023] (req 25468928->25473023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760790.596204:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484367104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 64067us (64309us total) trans 4294973536 rc 0/0 00000100:00100000:1.0:1789760790.596215:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6247 00000100:00100000:1.0:1789760790.596226:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779807776696 time 779863051716 state 0x2 ref 0 00000100:00100000:1.0:1789760790.599863:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.599972:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484367232 00000100:00100000:3.0:1789760790.599982:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484367232 00000100:00100000:3.0:1789760790.600036:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779807776696 time 779863051716 state 0x0 ref 1 00000100:00100000:3.0:1789760790.600235:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779863312132 time 779863051716 state 0x1 ref 1 00000100:00100000:3.0:1789760790.600306:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779863312132 time 779931236860 state 0x0 ref 1 00000100:00100000:3.0:1789760790.600319:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484367232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.600380:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25473024, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760790.600389:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.600404:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:3.0:1789760790.600408:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.600414:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25473024->25477119] (req 25473024->25477119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.600427:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25473024->25477119] (req 25473024->25477119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760790.628957:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973537 is committed 00002000:00010000:3.0:1789760790.629175:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25473024 -> 25477120 00010000:00010000:3.0:1789760790.629188:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25473024->25477119] (req 25473024->25477119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.629206:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25473024->25477119] (req 25473024->25477119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.629214:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25473024->25477119] (req 25473024->25477119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.629220:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760790.629225:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25473024->25477119] (req 25473024->25477119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.629233:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25473024->25477119] (req 25473024->25477119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.629251:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25473024->25477119] (req 25473024->25477119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.629259:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3ca lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25473024->25477119] (req 25473024->25477119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760790.629338:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484367232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29017us (29480us total) trans 4294973537 rc 0/0 00000100:00100000:3.0:1789760790.629348:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6248 00000100:00100000:3.0:1789760790.629357:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779863312132 time 779931236860 state 0x2 ref 0 00000100:00100000:3.0:1789760790.634195:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760790.634456:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484367360 00000100:00100000:0.0:1789760790.634466:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484367360 00000100:00100000:0.0:1789760790.634523:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779863312132 time 779931236860 state 0x0 ref 1 00000100:00100000:0.0:1789760790.634538:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779931497276 time 779931236860 state 0x1 ref 1 00000100:00100000:0.0:1789760790.634565:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779931497276 time 779965496651 state 0x0 ref 1 00000100:00100000:0.0:1789760790.634583:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484367360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760790.634629:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25477120, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:0.0:1789760790.634638:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.634654:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=779s v=(0 0) 00010000:00010000:0.0:1789760790.634658:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760790.634665:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25477120->25481215] (req 25477120->25481215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.634679:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25477120->25481215] (req 25477120->25481215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760790.667812:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973538 is committed 00002000:00010000:0.0:1789760790.668122:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25477120 -> 25481216 00010000:00010000:0.0:1789760790.668136:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25477120->25481215] (req 25477120->25481215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.668155:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25477120->25481215] (req 25477120->25481215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.668164:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25477120->25481215] (req 25477120->25481215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.668171:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760790.668175:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25477120->25481215] (req 25477120->25481215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.668184:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25477120->25481215] (req 25477120->25481215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.668201:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25477120->25481215] (req 25477120->25481215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.668209:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3d1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25477120->25481215] (req 25477120->25481215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760790.668379:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484367360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33794us (34188us total) trans 4294973538 rc 0/0 00000100:00100000:0.0:1789760790.668388:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6249 00000100:00100000:0.0:1789760790.668398:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779931497276 time 779965496651 state 0x2 ref 0 00000100:00100000:1.0:1789760790.671318:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760790.671388:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484367488 00000100:00100000:2.0:1789760790.671397:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484367488 00000100:00100000:2.0:1789760790.671444:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779931497276 time 779965496651 state 0x0 ref 1 00000100:00100000:2.0:1789760790.671457:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779965757067 time 779965496651 state 0x1 ref 1 00000100:00100000:2.0:1789760790.671486:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779965757067 time 780002417404 state 0x0 ref 1 00000100:00100000:2.0:1789760790.671498:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484367488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760790.671544:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25481216, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:2.0:1789760790.671554:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.671569:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:2.0:1789760790.671573:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760790.671579:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25481216->25485311] (req 25481216->25485311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.671594:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25481216->25485311] (req 25481216->25485311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760790.705938:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973539 is committed 00002000:00010000:3.0:1789760790.706136:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25481216 -> 25485312 00010000:00010000:3.0:1789760790.706151:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25481216->25485311] (req 25481216->25485311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.706165:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25481216->25485311] (req 25481216->25485311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.706175:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25481216->25485311] (req 25481216->25485311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.706183:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760790.706188:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25481216->25485311] (req 25481216->25485311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.706198:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25481216->25485311] (req 25481216->25485311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.706215:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25481216->25485311] (req 25481216->25485311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.706224:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a3d8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25481216->25485311] (req 25481216->25485311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760790.706349:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484367488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34848us (35035us total) trans 4294973539 rc 0/0 00000100:00100000:3.0:1789760790.706359:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6250 00000100:00100000:3.0:1789760790.706369:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779965757067 time 780002417404 state 0x2 ref 0 00000100:00100000:2.0:1789760790.709628:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760790.709703:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484367616 00000100:00100000:0.0:1789760790.709712:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484367616 00000100:00100000:0.0:1789760790.709813:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 779965757067 time 780002417404 state 0x0 ref 1 00000100:00100000:0.0:1789760790.709829:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780002677820 time 780002417404 state 0x1 ref 1 00000100:00100000:0.0:1789760790.709885:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780002677820 time 780040816040 state 0x0 ref 1 00000100:00100000:0.0:1789760790.709898:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484367616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760790.709941:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25485312, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:0.0:1789760790.709959:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.709976:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:0.0:1789760790.709980:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760790.709986:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25485312->25489407] (req 25485312->25489407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.710000:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25485312->25489407] (req 25485312->25489407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760790.736456:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973540 is committed 00002000:00010000:0.0:1789760790.736879:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25485312 -> 25489408 00010000:00010000:0.0:1789760790.736894:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25485312->25489407] (req 25485312->25489407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.736908:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25485312->25489407] (req 25485312->25489407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.736925:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25485312->25489407] (req 25485312->25489407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.736934:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760790.736939:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25485312->25489407] (req 25485312->25489407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.736957:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25485312->25489407] (req 25485312->25489407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.736978:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25485312->25489407] (req 25485312->25489407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.736986:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3df lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25485312->25489407] (req 25485312->25489407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760790.739263:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484367616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29354us (29630us total) trans 4294973540 rc 0/0 00000100:00100000:0.0:1789760790.739276:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6251 00000100:00100000:0.0:1789760790.739286:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780002677820 time 780040816040 state 0x2 ref 0 00000100:00100000:1.0:1789760790.742665:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.742751:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484367744 00000100:00100000:3.0:1789760790.742761:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484367744 00000100:00100000:3.0:1789760790.742811:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780002677820 time 780040816040 state 0x0 ref 1 00000100:00100000:3.0:1789760790.742826:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780041076456 time 780040816040 state 0x1 ref 1 00000100:00100000:3.0:1789760790.742910:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780041076456 time 780073840268 state 0x0 ref 1 00000100:00100000:3.0:1789760790.742924:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484367744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.742984:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25489408, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760790.742992:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.743009:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:3.0:1789760790.743013:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.743018:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25489408->25493503] (req 25489408->25493503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.743031:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25489408->25493503] (req 25489408->25493503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760790.776408:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973541 is committed 00002000:00010000:1.0:1789760790.776481:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25489408 -> 25493504 00010000:00010000:1.0:1789760790.776497:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25489408->25493503] (req 25489408->25493503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.776511:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25489408->25493503] (req 25489408->25493503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.776520:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25489408->25493503] (req 25489408->25493503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.776528:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760790.776533:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25489408->25493503] (req 25489408->25493503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.776542:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25489408->25493503] (req 25489408->25493503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.776563:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25489408->25493503] (req 25489408->25493503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.776571:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a3e6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25489408->25493503] (req 25489408->25493503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760790.776667:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484367744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33739us (34004us total) trans 4294973541 rc 0/0 00000100:00100000:1.0:1789760790.776686:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6252 00000100:00100000:1.0:1789760790.776695:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780041076456 time 780073840268 state 0x2 ref 0 00000100:00100000:2.0:1789760790.780823:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.780923:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484367872 00000100:00100000:3.0:1789760790.780932:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484367872 00000100:00100000:3.0:1789760790.781000:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780041076456 time 780073840268 state 0x0 ref 1 00000100:00100000:3.0:1789760790.781016:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780074100684 time 780073840268 state 0x1 ref 1 00000100:00100000:3.0:1789760790.781060:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780074100684 time 780111990892 state 0x0 ref 1 00000100:00100000:3.0:1789760790.781105:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484367872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.781152:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25493504, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760790.781161:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.781175:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:3.0:1789760790.781179:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.781185:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25493504->25497599] (req 25493504->25497599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.781199:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25493504->25497599] (req 25493504->25497599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760790.817857:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973542 is committed 00002000:00010000:2.0:1789760790.817984:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25493504 -> 25497600 00010000:00010000:2.0:1789760790.818002:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25493504->25497599] (req 25493504->25497599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.818015:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25493504->25497599] (req 25493504->25497599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.818023:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25493504->25497599] (req 25493504->25497599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.818031:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760790.818046:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25493504->25497599] (req 25493504->25497599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.818062:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25493504->25497599] (req 25493504->25497599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.818106:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25493504->25497599] (req 25493504->25497599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.818115:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a3ed lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25493504->25497599] (req 25493504->25497599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760790.818218:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484367872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37109us (37397us total) trans 4294973542 rc 0/0 00000100:00100000:2.0:1789760790.818227:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6253 00000100:00100000:2.0:1789760790.818237:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780074100684 time 780111990892 state 0x2 ref 0 00000100:00100000:2.0:1789760790.821313:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.821374:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484368000 00000100:00100000:3.0:1789760790.821384:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484368000 00000100:00100000:3.0:1789760790.821438:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780074100684 time 780111990892 state 0x0 ref 1 00000100:00100000:3.0:1789760790.821453:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780112251308 time 780111990892 state 0x1 ref 1 00000100:00100000:3.0:1789760790.821480:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780112251308 time 780152411616 state 0x0 ref 1 00000100:00100000:3.0:1789760790.821491:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484368000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.821542:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25497600, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760790.821551:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.821566:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:3.0:1789760790.821570:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.821576:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25497600->25501695] (req 25497600->25501695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.821589:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25497600->25501695] (req 25497600->25501695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760790.846350:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25497600 -> 25501696 00010000:00010000:2.0:1789760790.846371:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25497600->25501695] (req 25497600->25501695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.846383:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25497600->25501695] (req 25497600->25501695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.846391:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25497600->25501695] (req 25497600->25501695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.846403:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760790.846407:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25497600->25501695] (req 25497600->25501695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.846415:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25497600->25501695] (req 25497600->25501695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.846434:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25497600->25501695] (req 25497600->25501695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760790.846442:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a3f4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25497600->25501695] (req 25497600->25501695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760790.846528:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484368000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25034us (25219us total) trans 4294973543 rc 0/0 00000100:00100000:2.0:1789760790.846538:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6254 00000100:00100000:2.0:1789760790.846547:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780112251308 time 780152411616 state 0x2 ref 0 00000001:00080000:1.0:1789760790.846943:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973543 is committed 00000100:00100000:1.0:1789760790.850594:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760790.850826:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484368128 00000100:00100000:0.0:1789760790.850836:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484368128 00000100:00100000:0.0:1789760790.850891:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780112251308 time 780152411616 state 0x0 ref 1 00000100:00100000:0.0:1789760790.850903:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780152672032 time 780152411616 state 0x1 ref 1 00000100:00100000:0.0:1789760790.850961:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780152672032 time 780181891187 state 0x0 ref 1 00000100:00100000:0.0:1789760790.850983:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484368128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760790.851026:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25501696, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:0.0:1789760790.851032:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.851045:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:0.0:1789760790.851049:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760790.851055:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25501696->25505791] (req 25501696->25505791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.851099:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25501696->25505791] (req 25501696->25505791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760790.880675:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973544 is committed 00002000:00010000:1.0:1789760790.880852:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25501696 -> 25505792 00010000:00010000:1.0:1789760790.880868:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25501696->25505791] (req 25501696->25505791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.880881:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25501696->25505791] (req 25501696->25505791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.880889:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25501696->25505791] (req 25501696->25505791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.880897:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760790.880903:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25501696->25505791] (req 25501696->25505791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.880911:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25501696->25505791] (req 25501696->25505791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.880929:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25501696->25505791] (req 25501696->25505791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.880937:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a3fb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25501696->25505791] (req 25501696->25505791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760790.881233:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484368128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30247us (30641us total) trans 4294973544 rc 0/0 00000100:00100000:1.0:1789760790.881243:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6255 00000100:00100000:1.0:1789760790.881252:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780152672032 time 780181891187 state 0x2 ref 0 00000100:00100000:3.0:1789760790.884314:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760790.884404:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484368256 00000100:00100000:1.0:1789760790.884413:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484368256 00000100:00100000:1.0:1789760790.884461:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780152672032 time 780181891187 state 0x0 ref 1 00000100:00100000:1.0:1789760790.884476:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780182151603 time 780181891187 state 0x1 ref 1 00000100:00100000:1.0:1789760790.884549:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780182151603 time 780215479667 state 0x0 ref 1 00000100:00100000:1.0:1789760790.884562:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484368256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760790.884608:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25505792, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:1.0:1789760790.884617:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.884630:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:1.0:1789760790.884634:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760790.884640:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25505792->25509887] (req 25505792->25509887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.884659:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25505792->25509887] (req 25505792->25509887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760790.917595:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973545 is committed 00002000:00010000:1.0:1789760790.917687:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25505792 -> 25509888 00010000:00010000:1.0:1789760790.917700:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25505792->25509887] (req 25505792->25509887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.917710:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25505792->25509887] (req 25505792->25509887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.917719:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25505792->25509887] (req 25505792->25509887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.917727:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760790.917793:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25505792->25509887] (req 25505792->25509887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.917803:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25505792->25509887] (req 25505792->25509887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.917821:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25505792->25509887] (req 25505792->25509887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.917828:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a402 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25505792->25509887] (req 25505792->25509887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760790.917963:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484368256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33398us (33653us total) trans 4294973545 rc 0/0 00000100:00100000:1.0:1789760790.917974:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6256 00000100:00100000:1.0:1789760790.917983:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780182151603 time 780215479667 state 0x2 ref 0 00000100:00100000:2.0:1789760790.921402:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760790.921481:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484368384 00000100:00100000:3.0:1789760790.921491:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484368384 00000100:00100000:3.0:1789760790.921539:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780182151603 time 780215479667 state 0x0 ref 1 00000100:00100000:3.0:1789760790.921554:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780215740083 time 780215479667 state 0x1 ref 1 00000100:00100000:3.0:1789760790.921584:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780215740083 time 780252513970 state 0x0 ref 1 00000100:00100000:3.0:1789760790.921596:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484368384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760790.921643:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25509888, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760790.921651:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.921666:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:3.0:1789760790.921670:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760790.921675:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25509888->25513983] (req 25509888->25513983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760790.921687:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25509888->25513983] (req 25509888->25513983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760790.954633:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25509888 -> 25513984 00010000:00010000:0.0:1789760790.954652:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25509888->25513983] (req 25509888->25513983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.954665:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25509888->25513983] (req 25509888->25513983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.954673:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25509888->25513983] (req 25509888->25513983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.954681:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760790.954686:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25509888->25513983] (req 25509888->25513983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.954695:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25509888->25513983] (req 25509888->25513983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.954714:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25509888->25513983] (req 25509888->25513983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.954721:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a409 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25509888->25513983] (req 25509888->25513983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760790.954822:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82ad80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484368384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33222us (33422us total) trans 4294973546 rc 0/0 00000100:00100000:0.0:1789760790.954832:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6257 00000100:00100000:0.0:1789760790.954842:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780215740083 time 780252513970 state 0x2 ref 0 00000001:00080000:1.0:1789760790.954969:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973546 is committed 00000100:00100000:3.0:1789760790.957918:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760790.958055:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484368512 00000100:00100000:0.0:1789760790.958084:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484368512 00000100:00100000:0.0:1789760790.958145:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780215740083 time 780252513970 state 0x0 ref 1 00000100:00100000:0.0:1789760790.958161:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780252774386 time 780252513970 state 0x1 ref 1 00000100:00100000:0.0:1789760790.958190:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780252774386 time 780289121050 state 0x0 ref 1 00000100:00100000:0.0:1789760790.958204:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484368512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760790.958250:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25513984, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:0.0:1789760790.958258:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.958273:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:0.0:1789760790.958277:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760790.958282:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25513984->25518079] (req 25513984->25518079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.958295:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25513984->25518079] (req 25513984->25518079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760790.982941:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973547 is committed 00002000:00010000:0.0:1789760790.983164:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25513984 -> 25518080 00010000:00010000:0.0:1789760790.983181:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25513984->25518079] (req 25513984->25518079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.983195:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25513984->25518079] (req 25513984->25518079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.983204:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25513984->25518079] (req 25513984->25518079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.983212:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760790.983217:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25513984->25518079] (req 25513984->25518079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.983225:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25513984->25518079] (req 25513984->25518079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.983251:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25513984->25518079] (req 25513984->25518079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760790.983259:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a410 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25513984->25518079] (req 25513984->25518079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760790.983390:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484368512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25184us (25476us total) trans 4294973547 rc 0/0 00000100:00100000:0.0:1789760790.983400:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6258 00000100:00100000:0.0:1789760790.983410:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780252774386 time 780289121050 state 0x2 ref 0 00000100:00100000:3.0:1789760790.987898:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760790.987974:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484368640 00000100:00100000:1.0:1789760790.987985:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484368640 00000100:00100000:1.0:1789760790.988052:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780252774386 time 780289121050 state 0x0 ref 1 00000100:00100000:1.0:1789760790.988092:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780289381466 time 780289121050 state 0x1 ref 1 00000100:00100000:1.0:1789760790.988115:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780289381466 time 780319046633 state 0x0 ref 1 00000100:00100000:1.0:1789760790.988128:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484368640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760790.988174:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25518080, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:1.0:1789760790.988182:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.988197:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:1.0:1789760790.988201:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760790.988207:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25518080->25522175] (req 25518080->25522175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760790.988228:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25518080->25522175] (req 25518080->25522175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760791.014613:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25518080 -> 25522176 00010000:00010000:1.0:1789760791.014636:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25518080->25522175] (req 25518080->25522175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.014648:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25518080->25522175] (req 25518080->25522175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.014656:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25518080->25522175] (req 25518080->25522175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.014664:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.014670:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25518080->25522175] (req 25518080->25522175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.014678:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25518080->25522175] (req 25518080->25522175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.014697:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25518080->25522175] (req 25518080->25522175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.014705:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a417 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25518080->25522175] (req 25518080->25522175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.014811:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484368640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26681us (26919us total) trans 4294973548 rc 0/0 00000100:00100000:1.0:1789760791.014825:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6259 00000100:00100000:1.0:1789760791.014835:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780289381466 time 780319046633 state 0x2 ref 0 00000001:00080000:2.0:1789760791.014974:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973548 is committed 00000100:00100000:2.0:1789760791.018202:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760791.018388:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484368768 00000100:00100000:1.0:1789760791.018399:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484368768 00000100:00100000:1.0:1789760791.018451:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780289381466 time 780319046633 state 0x0 ref 1 00000100:00100000:1.0:1789760791.018466:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780319307049 time 780319046633 state 0x1 ref 1 00000100:00100000:1.0:1789760791.018530:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780319307049 time 780349459973 state 0x0 ref 1 00000100:00100000:1.0:1789760791.018544:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484368768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760791.018592:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25522176, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:1.0:1789760791.018601:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.018614:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:1.0:1789760791.018618:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760791.018624:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25522176->25526271] (req 25522176->25526271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.018638:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25522176->25526271] (req 25522176->25526271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760791.042647:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973549 is committed 00002000:00010000:1.0:1789760791.042762:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25522176 -> 25526272 00010000:00010000:1.0:1789760791.042777:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25522176->25526271] (req 25522176->25526271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.042792:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25522176->25526271] (req 25522176->25526271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.042802:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25522176->25526271] (req 25522176->25526271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.042809:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.042814:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25522176->25526271] (req 25522176->25526271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.042824:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25522176->25526271] (req 25522176->25526271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.042841:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25522176->25526271] (req 25522176->25526271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.042849:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a41e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25522176->25526271] (req 25522176->25526271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.042931:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484368768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24386us (24735us total) trans 4294973549 rc 0/0 00000100:00100000:1.0:1789760791.042940:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6260 00000100:00100000:1.0:1789760791.042959:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780319307049 time 780349459973 state 0x2 ref 0 00000100:00100000:3.0:1789760791.045877:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760791.050049:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484368896 00000100:00100000:1.0:1789760791.050087:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484368896 00000100:00100000:1.0:1789760791.050146:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780319307049 time 780349459973 state 0x0 ref 1 00000100:00100000:1.0:1789760791.050161:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780349720389 time 780349459973 state 0x1 ref 1 00000100:00100000:1.0:1789760791.050186:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780349720389 time 780381117199 state 0x0 ref 1 00000100:00100000:1.0:1789760791.050197:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484368896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760791.050252:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25526272, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:1.0:1789760791.050261:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.050277:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:1.0:1789760791.050281:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760791.050288:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25526272->25530367] (req 25526272->25530367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.050302:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25526272->25530367] (req 25526272->25530367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760791.080697:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973550 is committed 00002000:00010000:1.0:1789760791.080848:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25526272 -> 25530368 00010000:00010000:1.0:1789760791.080862:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25526272->25530367] (req 25526272->25530367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.080874:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25526272->25530367] (req 25526272->25530367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.080888:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25526272->25530367] (req 25526272->25530367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.080896:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.080900:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25526272->25530367] (req 25526272->25530367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.080909:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25526272->25530367] (req 25526272->25530367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.080927:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25526272->25530367] (req 25526272->25530367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.080935:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a425 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25526272->25530367] (req 25526272->25530367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.081031:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484368896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30829us (35158us total) trans 4294973550 rc 0/0 00000100:00100000:1.0:1789760791.081052:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6261 00000100:00100000:1.0:1789760791.081061:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780349720389 time 780381117199 state 0x2 ref 0 00000100:00100000:1.0:1789760791.084196:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760791.084284:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484369024 00000100:00100000:0.0:1789760791.084293:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484369024 00000100:00100000:0.0:1789760791.084349:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780349720389 time 780381117199 state 0x0 ref 1 00000100:00100000:0.0:1789760791.084366:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780381377615 time 780381117199 state 0x1 ref 1 00000100:00100000:0.0:1789760791.084421:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780381377615 time 780415351791 state 0x0 ref 1 00000100:00100000:0.0:1789760791.084435:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760791.084482:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25530368, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:0.0:1789760791.084491:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.084506:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:0.0:1789760791.084511:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760791.084517:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25530368->25534463] (req 25530368->25534463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.084529:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25530368->25534463] (req 25530368->25534463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760791.118109:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973551 is committed 00002000:00010000:2.0:1789760791.118381:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25530368 -> 25534464 00010000:00010000:2.0:1789760791.118395:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25530368->25534463] (req 25530368->25534463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.118406:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25530368->25534463] (req 25530368->25534463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.118414:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25530368->25534463] (req 25530368->25534463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.118421:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760791.118426:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25530368->25534463] (req 25530368->25534463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.118436:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25530368->25534463] (req 25530368->25534463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.118457:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25530368->25534463] (req 25530368->25534463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.118465:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a42c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25530368->25534463] (req 25530368->25534463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760791.118557:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34120us (34364us total) trans 4294973551 rc 0/0 00000100:00100000:2.0:1789760791.118566:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6262 00000100:00100000:2.0:1789760791.118576:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780381377615 time 780415351791 state 0x2 ref 0 00000100:00100000:1.0:1789760791.121319:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760791.121395:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484369152 00000100:00100000:3.0:1789760791.121405:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484369152 00000100:00100000:3.0:1789760791.121457:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780381377615 time 780415351791 state 0x0 ref 1 00000100:00100000:3.0:1789760791.121471:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780415612207 time 780415351791 state 0x1 ref 1 00000100:00100000:3.0:1789760791.121499:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780415612207 time 780452430127 state 0x0 ref 1 00000100:00100000:3.0:1789760791.121511:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760791.121556:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25534464, mtime: 0x6aad9516, atime: 0x0, ctime: 0x6aad9516, blocks: 0xc807 00010000:00010000:3.0:1789760791.121564:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.121579:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:3.0:1789760791.121584:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760791.121888:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25534464->25538559] (req 25534464->25538559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.121903:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25534464->25538559] (req 25534464->25538559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760791.148395:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25534464 -> 25538560 00002000:00010000:0.0:1789760791.148404:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760790 -> 1789760791 00002000:00010000:0.0:1789760791.148407:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760790 -> 1789760791 00010000:00010000:0.0:1789760791.148421:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25534464->25538559] (req 25534464->25538559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.148434:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25534464->25538559] (req 25534464->25538559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.148443:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25534464->25538559] (req 25534464->25538559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.148450:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760791.148455:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25534464->25538559] (req 25534464->25538559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.148465:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25534464->25538559] (req 25534464->25538559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.148486:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25534464->25538559] (req 25534464->25538559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.148495:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a433 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25534464->25538559] (req 25534464->25538559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760791.148587:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484369152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27072us (27271us total) trans 4294973552 rc 0/0 00000100:00100000:0.0:1789760791.148597:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6263 00000100:00100000:0.0:1789760791.148607:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780415612207 time 780452430127 state 0x2 ref 0 00000001:00080000:2.0:1789760791.149240:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973552 is committed 00000100:00100000:1.0:1789760791.153800:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760791.154435:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484369280 00000100:00100000:2.0:1789760791.154447:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484369280 00000100:00100000:2.0:1789760791.154502:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780415612207 time 780452430127 state 0x0 ref 1 00000100:00100000:2.0:1789760791.154517:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780452690543 time 780452430127 state 0x1 ref 1 00000100:00100000:2.0:1789760791.154581:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780452690543 time 780485511583 state 0x0 ref 1 00000100:00100000:2.0:1789760791.154593:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484369280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760791.154644:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25538560, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:2.0:1789760791.154654:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.154669:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:2.0:1789760791.154674:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760791.154680:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25538560->25542655] (req 25538560->25542655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.154694:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25538560->25542655] (req 25538560->25542655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760791.193030:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973553 is committed 00002000:00010000:2.0:1789760791.193168:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25538560 -> 25542656 00010000:00010000:2.0:1789760791.193183:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25538560->25542655] (req 25538560->25542655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.193195:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25538560->25542655] (req 25538560->25542655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.193209:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25538560->25542655] (req 25538560->25542655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.193216:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760791.193221:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25538560->25542655] (req 25538560->25542655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.193230:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25538560->25542655] (req 25538560->25542655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.193250:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25538560->25542655] (req 25538560->25542655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.193257:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a43a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25538560->25542655] (req 25538560->25542655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760791.193379:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484369280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38783us (39583us total) trans 4294973553 rc 0/0 00000100:00100000:2.0:1789760791.193390:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6264 00000100:00100000:2.0:1789760791.193399:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780452690543 time 780485511583 state 0x2 ref 0 00000100:00100000:3.0:1789760791.196896:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760791.197281:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484369408 00000100:00100000:1.0:1789760791.197291:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484369408 00000100:00100000:1.0:1789760791.197348:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780452690543 time 780485511583 state 0x0 ref 1 00000100:00100000:1.0:1789760791.197364:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780485771999 time 780485511583 state 0x1 ref 1 00000100:00100000:1.0:1789760791.197449:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780485771999 time 780528378463 state 0x0 ref 1 00000100:00100000:1.0:1789760791.197463:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760791.197510:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25542656, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:1.0:1789760791.197519:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.197535:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:1.0:1789760791.197540:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760791.197547:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25542656->25546751] (req 25542656->25546751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.197561:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25542656->25546751] (req 25542656->25546751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760791.230969:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973554 is committed 00002000:00010000:1.0:1789760791.231091:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25542656 -> 25546752 00010000:00010000:1.0:1789760791.231107:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25542656->25546751] (req 25542656->25546751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.231119:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25542656->25546751] (req 25542656->25546751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.231129:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25542656->25546751] (req 25542656->25546751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.231138:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.231143:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25542656->25546751] (req 25542656->25546751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.231153:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25542656->25546751] (req 25542656->25546751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.231175:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25542656->25546751] (req 25542656->25546751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.231184:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a441 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25542656->25546751] (req 25542656->25546751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.231310:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33843us (34418us total) trans 4294973554 rc 0/0 00000100:00100000:1.0:1789760791.231320:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6265 00000100:00100000:1.0:1789760791.231329:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780485771999 time 780528378463 state 0x2 ref 0 00000100:00100000:1.0:1789760791.249763:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760791.249850:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484369536 00000100:00100000:0.0:1789760791.249859:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484369536 00000100:00100000:0.0:1789760791.249915:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780485771999 time 780528378463 state 0x0 ref 1 00000100:00100000:0.0:1789760791.249930:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780528638879 time 780528378463 state 0x1 ref 1 00000100:00100000:0.0:1789760791.249967:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780528638879 time 780580898175 state 0x0 ref 1 00000100:00100000:0.0:1789760791.249980:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760791.250028:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25546752, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:0.0:1789760791.250036:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.250052:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:0.0:1789760791.250057:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760791.250093:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25546752->25550847] (req 25546752->25550847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.250114:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25546752->25550847] (req 25546752->25550847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760791.285415:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973555 is committed 00002000:00010000:1.0:1789760791.285567:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25546752 -> 25550848 00010000:00010000:1.0:1789760791.285583:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25546752->25550847] (req 25546752->25550847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.285602:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25546752->25550847] (req 25546752->25550847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.285611:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25546752->25550847] (req 25546752->25550847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.285619:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.285624:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25546752->25550847] (req 25546752->25550847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.285633:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25546752->25550847] (req 25546752->25550847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.285652:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25546752->25550847] (req 25546752->25550847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.285660:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a448 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25546752->25550847] (req 25546752->25550847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.285797:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35813us (36038us total) trans 4294973555 rc 0/0 00000100:00100000:1.0:1789760791.285807:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6266 00000100:00100000:1.0:1789760791.285826:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780528638879 time 780580898175 state 0x2 ref 0 00000100:00100000:3.0:1789760791.289609:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760791.289722:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484369664 00000100:00100000:0.0:1789760791.289779:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484369664 00000100:00100000:0.0:1789760791.289835:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780528638879 time 780580898175 state 0x0 ref 1 00000100:00100000:0.0:1789760791.289849:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780581158591 time 780580898175 state 0x1 ref 1 00000100:00100000:0.0:1789760791.289878:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780581158591 time 780620809267 state 0x0 ref 1 00000100:00100000:0.0:1789760791.289891:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760791.289936:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25550848, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:0.0:1789760791.289955:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.289969:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:0.0:1789760791.289973:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760791.289979:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25550848->25554943] (req 25550848->25554943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.289994:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25550848->25554943] (req 25550848->25554943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760791.312853:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973556 is committed 00002000:00010000:0.0:1789760791.313041:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25550848 -> 25554944 00010000:00010000:0.0:1789760791.313057:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25550848->25554943] (req 25550848->25554943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.313106:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25550848->25554943] (req 25550848->25554943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.313115:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25550848->25554943] (req 25550848->25554943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.313122:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760791.313127:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25550848->25554943] (req 25550848->25554943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.313137:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25550848->25554943] (req 25550848->25554943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.313156:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25550848->25554943] (req 25550848->25554943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.313164:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a44f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25550848->25554943] (req 25550848->25554943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760791.313247:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 23354us (23642us total) trans 4294973556 rc 0/0 00000100:00100000:0.0:1789760791.313257:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6267 00000100:00100000:0.0:1789760791.313267:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780581158591 time 780620809267 state 0x2 ref 0 00000100:00100000:3.0:1789760791.317498:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760791.317833:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484369792 00000100:00100000:2.0:1789760791.317843:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484369792 00000100:00100000:2.0:1789760791.317965:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780581158591 time 780620809267 state 0x0 ref 1 00000100:00100000:2.0:1789760791.317981:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780621069683 time 780620809267 state 0x1 ref 1 00000100:00100000:2.0:1789760791.318047:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780621069683 time 780648977021 state 0x0 ref 1 00000100:00100000:2.0:1789760791.318060:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760791.318138:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25554944, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:2.0:1789760791.318147:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.318162:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:2.0:1789760791.318167:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760791.318173:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25554944->25559039] (req 25554944->25559039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.318187:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25554944->25559039] (req 25554944->25559039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760791.344803:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25554944 -> 25559040 00010000:00010000:2.0:1789760791.344825:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25554944->25559039] (req 25554944->25559039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.344838:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25554944->25559039] (req 25554944->25559039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.344847:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25554944->25559039] (req 25554944->25559039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.344854:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760791.344860:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25554944->25559039] (req 25554944->25559039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.344869:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25554944->25559039] (req 25554944->25559039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.344889:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25554944->25559039] (req 25554944->25559039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.344897:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a456 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25554944->25559039] (req 25554944->25559039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760791.344995:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484369792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26933us (27503us total) trans 4294973557 rc 0/0 00000100:00100000:2.0:1789760791.345005:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6268 00000100:00100000:2.0:1789760791.345014:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780621069683 time 780648977021 state 0x2 ref 0 00000001:00080000:3.0:1789760791.347397:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973557 is committed 00000100:00100000:2.0:1789760791.348543:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760791.348636:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484369920 00000100:00100000:1.0:1789760791.348648:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484369920 00000100:00100000:1.0:1789760791.348701:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780621069683 time 780648977021 state 0x0 ref 1 00000100:00100000:1.0:1789760791.348717:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780649237437 time 780648977021 state 0x1 ref 1 00000100:00100000:1.0:1789760791.348755:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780649237437 time 780679685828 state 0x0 ref 1 00000100:00100000:1.0:1789760791.348768:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484369920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760791.348816:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25559040, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:1.0:1789760791.348824:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.348840:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:1.0:1789760791.348844:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760791.348851:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25559040->25563135] (req 25559040->25563135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.348865:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25559040->25563135] (req 25559040->25563135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760791.377712:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25559040 -> 25563136 00010000:00010000:1.0:1789760791.377743:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25559040->25563135] (req 25559040->25563135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.377756:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25559040->25563135] (req 25559040->25563135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.377770:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25559040->25563135] (req 25559040->25563135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.377777:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.377783:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25559040->25563135] (req 25559040->25563135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.377791:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25559040->25563135] (req 25559040->25563135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.377809:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25559040->25563135] (req 25559040->25563135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.377817:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a45d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25559040->25563135] (req 25559040->25563135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.377908:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484369920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29137us (29368us total) trans 4294973558 rc 0/0 00000100:00100000:1.0:1789760791.377918:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6269 00000100:00100000:1.0:1789760791.377927:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780649237437 time 780679685828 state 0x2 ref 0 00000001:00080000:2.0:1789760791.380276:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973558 is committed 00000100:00100000:1.0:1789760791.381637:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760791.381702:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484370048 00000100:00100000:0.0:1789760791.381712:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484370048 00000100:00100000:0.0:1789760791.381817:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780649237437 time 780679685828 state 0x0 ref 1 00000100:00100000:0.0:1789760791.381832:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780679946244 time 780679685828 state 0x1 ref 1 00000100:00100000:0.0:1789760791.381860:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780679946244 time 780712791522 state 0x0 ref 1 00000100:00100000:0.0:1789760791.381875:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb726680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484370048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760791.382239:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25563136, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:0.0:1789760791.382249:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.382264:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:0.0:1789760791.382268:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760791.382275:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25563136->25567231] (req 25563136->25567231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.382287:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25563136->25567231] (req 25563136->25567231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760791.404121:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25563136 -> 25567232 00010000:00010000:0.0:1789760791.404140:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25563136->25567231] (req 25563136->25567231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.404154:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25563136->25567231] (req 25563136->25567231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.404164:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25563136->25567231] (req 25563136->25567231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.404171:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760791.404176:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25563136->25567231] (req 25563136->25567231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.404186:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25563136->25567231] (req 25563136->25567231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.404205:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25563136->25567231] (req 25563136->25567231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.404213:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a464 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25563136->25567231] (req 25563136->25567231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760791.404298:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb726680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484370048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 22423us (22664us total) trans 4294973559 rc 0/0 00000100:00100000:0.0:1789760791.404308:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6270 00000100:00100000:0.0:1789760791.404317:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780679946244 time 780712791522 state 0x2 ref 0 00000100:00100000:3.0:1789760791.407593:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760791.407673:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484370176 00000100:00100000:0.0:1789760791.407683:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484370176 00000100:00100000:0.0:1789760791.407748:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780679946244 time 780712791522 state 0x0 ref 1 00000100:00100000:0.0:1789760791.407763:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780713051938 time 780712791522 state 0x1 ref 1 00000100:00100000:0.0:1789760791.407799:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780713051938 time 780738729261 state 0x0 ref 1 00000100:00100000:0.0:1789760791.407812:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484370176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760791.407860:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25567232, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:0.0:1789760791.407869:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.407885:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:0.0:1789760791.407890:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760791.407896:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25567232->25571327] (req 25567232->25571327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.407909:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25567232->25571327] (req 25567232->25571327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760791.412544:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973559 is committed 00010000:00010000:1.0:1789760791.436248:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760791.439622:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973560 is committed 00002000:00010000:1.0:1789760791.439741:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25567232 -> 25571328 00010000:00010000:1.0:1789760791.439757:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25567232->25571327] (req 25567232->25571327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.439769:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25567232->25571327] (req 25567232->25571327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.439778:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25567232->25571327] (req 25567232->25571327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.439786:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.439791:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25567232->25571327] (req 25567232->25571327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.439799:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25567232->25571327] (req 25567232->25571327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.439819:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25567232->25571327] (req 25567232->25571327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.439827:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a46b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25567232->25571327] (req 25567232->25571327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.439919:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484370176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32106us (32330us total) trans 4294973560 rc 0/0 00000100:00100000:1.0:1789760791.439929:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6271 00000100:00100000:1.0:1789760791.439939:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780713051938 time 780738729261 state 0x2 ref 0 00000100:00100000:3.0:1789760791.443544:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760791.443647:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484370304 00000100:00100000:2.0:1789760791.443658:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484370304 00000100:00100000:2.0:1789760791.443709:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780713051938 time 780738729261 state 0x0 ref 1 00000100:00100000:2.0:1789760791.443735:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780738989677 time 780738729261 state 0x1 ref 1 00000100:00100000:2.0:1789760791.443801:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780738989677 time 780774732043 state 0x0 ref 1 00000100:00100000:2.0:1789760791.443814:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484370304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760791.443862:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25571328, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:2.0:1789760791.443870:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.443886:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:2.0:1789760791.443891:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760791.443897:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25571328->25575423] (req 25571328->25575423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.443911:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25571328->25575423] (req 25571328->25575423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760791.469521:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25571328 -> 25575424 00010000:00010000:1.0:1789760791.469541:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25571328->25575423] (req 25571328->25575423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.469556:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25571328->25575423] (req 25571328->25575423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.469566:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25571328->25575423] (req 25571328->25575423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.469574:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.469578:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25571328->25575423] (req 25571328->25575423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.469588:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25571328->25575423] (req 25571328->25575423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.469610:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25571328->25575423] (req 25571328->25575423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.469618:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a472 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25571328->25575423] (req 25571328->25575423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.469754:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484370304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 25937us (26215us total) trans 4294973561 rc 0/0 00000100:00100000:1.0:1789760791.469765:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6272 00000100:00100000:1.0:1789760791.469776:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780738989677 time 780774732043 state 0x2 ref 0 00000001:00080000:3.0:1789760791.469891:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973561 is committed 00000100:00100000:1.0:1789760791.474575:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760791.474672:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484370432 00000100:00100000:2.0:1789760791.474682:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484370432 00000100:00100000:2.0:1789760791.474888:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780738989677 time 780774732043 state 0x0 ref 1 00000100:00100000:2.0:1789760791.474904:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780774992459 time 780774732043 state 0x1 ref 1 00000100:00100000:2.0:1789760791.474968:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780774992459 time 780805898679 state 0x0 ref 1 00000100:00100000:2.0:1789760791.474982:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb727b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484370432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760791.475045:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25575424, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:2.0:1789760791.475054:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.475090:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:2.0:1789760791.475096:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760791.475103:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25575424->25579519] (req 25575424->25579519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.475118:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25575424->25579519] (req 25575424->25579519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760791.503542:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25575424 -> 25579520 00010000:00010000:3.0:1789760791.503564:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25575424->25579519] (req 25575424->25579519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.503578:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25575424->25579519] (req 25575424->25579519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.503587:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25575424->25579519] (req 25575424->25579519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.503598:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760791.503602:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25575424->25579519] (req 25575424->25579519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.503611:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25575424->25579519] (req 25575424->25579519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.503629:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25575424->25579519] (req 25575424->25579519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.503637:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a479 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25575424->25579519] (req 25575424->25579519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760791.503736:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb727b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484370432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28746us (29160us total) trans 4294973562 rc 0/0 00000100:00100000:3.0:1789760791.503747:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6273 00000100:00100000:3.0:1789760791.503756:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780774992459 time 780805898679 state 0x2 ref 0 00000001:00080000:3.0:1789760791.503793:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973562 is committed 00000100:00100000:0.0:1789760791.508802:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760791.508976:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484370560 00000100:00100000:1.0:1789760791.508987:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484370560 00000100:00100000:1.0:1789760791.509039:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780774992459 time 780805898679 state 0x0 ref 1 00000100:00100000:1.0:1789760791.509053:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780806159095 time 780805898679 state 0x1 ref 1 00000100:00100000:1.0:1789760791.509127:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780806159095 time 780840057352 state 0x0 ref 1 00000100:00100000:1.0:1789760791.509142:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484370560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760791.509190:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25579520, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:1.0:1789760791.509199:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.509215:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:1.0:1789760791.509219:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760791.509224:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25579520->25583615] (req 25579520->25583615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.509238:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25579520->25583615] (req 25579520->25583615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760791.548558:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973563 is committed 00002000:00010000:0.0:1789760791.548655:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25579520 -> 25583616 00010000:00010000:0.0:1789760791.548671:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25579520->25583615] (req 25579520->25583615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.548684:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25579520->25583615] (req 25579520->25583615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.548693:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25579520->25583615] (req 25579520->25583615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.548704:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760791.548709:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25579520->25583615] (req 25579520->25583615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.548717:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25579520->25583615] (req 25579520->25583615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.548805:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25579520->25583615] (req 25579520->25583615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.548814:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a480 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25579520->25583615] (req 25579520->25583615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760791.548918:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484370560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39774us (40121us total) trans 4294973563 rc 0/0 00000100:00100000:0.0:1789760791.548935:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6274 00000100:00100000:0.0:1789760791.548956:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780806159095 time 780840057352 state 0x2 ref 0 00000100:00100000:2.0:1789760791.557582:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760791.557686:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484370688 00000100:00100000:1.0:1789760791.557696:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484370688 00000100:00100000:1.0:1789760791.557804:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780806159095 time 780840057352 state 0x0 ref 1 00000100:00100000:1.0:1789760791.557825:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780840317768 time 780840057352 state 0x1 ref 1 00000100:00100000:1.0:1789760791.557854:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780840317768 time 780888785148 state 0x0 ref 1 00000100:00100000:1.0:1789760791.557869:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484370688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760791.557918:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25583616, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:1.0:1789760791.557927:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.557953:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:1.0:1789760791.557959:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760791.557966:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25583616->25587711] (req 25583616->25587711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.557980:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25583616->25587711] (req 25583616->25587711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760791.609236:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25583616 -> 25587712 00010000:00010000:0.0:1789760791.609370:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25583616->25587711] (req 25583616->25587711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760791.609456:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973564 is committed 00010000:00010000:0.0:1789760791.609577:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25583616->25587711] (req 25583616->25587711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.609595:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25583616->25587711] (req 25583616->25587711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.609602:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760791.609607:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25583616->25587711] (req 25583616->25587711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.609617:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25583616->25587711] (req 25583616->25587711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.609636:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25583616->25587711] (req 25583616->25587711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.609643:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a487 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25583616->25587711] (req 25583616->25587711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760791.609745:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484370688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 51874us (52290us total) trans 4294973564 rc 0/0 00000100:00100000:0.0:1789760791.609756:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6275 00000100:00100000:0.0:1789760791.609767:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780840317768 time 780888785148 state 0x2 ref 0 00000100:00100000:1.0:1789760791.613505:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760791.613684:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484370816 00000100:00100000:2.0:1789760791.613694:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484370816 00000100:00100000:2.0:1789760791.613749:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780840317768 time 780888785148 state 0x0 ref 1 00000100:00100000:2.0:1789760791.613765:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780889045564 time 780888785148 state 0x1 ref 1 00000100:00100000:2.0:1789760791.613788:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780889045564 time 780944719902 state 0x0 ref 1 00000100:00100000:2.0:1789760791.613802:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484370816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760791.613850:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25587712, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:2.0:1789760791.613860:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.613876:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=780s v=(0 0) 00010000:00010000:2.0:1789760791.613881:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760791.613887:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25587712->25591807] (req 25587712->25591807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.613901:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25587712->25591807] (req 25587712->25591807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760791.676166:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973565 is committed 00002000:00010000:1.0:1789760791.676280:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25587712 -> 25591808 00010000:00010000:1.0:1789760791.676296:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25587712->25591807] (req 25587712->25591807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.676310:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25587712->25591807] (req 25587712->25591807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.676320:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25587712->25591807] (req 25587712->25591807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.676329:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.676334:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25587712->25591807] (req 25587712->25591807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.676343:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25587712->25591807] (req 25587712->25591807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.676366:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25587712->25591807] (req 25587712->25591807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.676374:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a48e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25587712->25591807] (req 25587712->25591807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.677275:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484370816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63468us (63772us total) trans 4294973565 rc 0/0 00000100:00100000:1.0:1789760791.677288:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6276 00000100:00100000:1.0:1789760791.677298:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780889045564 time 780944719902 state 0x2 ref 0 00000100:00100000:3.0:1789760791.689851:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760791.690523:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484370944 00000100:00100000:2.0:1789760791.690533:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484370944 00000100:00100000:2.0:1789760791.690589:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780889045564 time 780944719902 state 0x0 ref 1 00000100:00100000:2.0:1789760791.690605:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780944980318 time 780944719902 state 0x1 ref 1 00000100:00100000:2.0:1789760791.690635:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780944980318 time 781021566530 state 0x0 ref 1 00000100:00100000:2.0:1789760791.690648:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484370944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760791.690695:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25591808, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:2.0:1789760791.690704:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.690719:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:2.0:1789760791.690734:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760791.690741:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25591808->25595903] (req 25591808->25595903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.690755:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25591808->25595903] (req 25591808->25595903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760791.752880:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973566 is committed 00002000:00010000:2.0:1789760791.752994:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25591808 -> 25595904 00010000:00010000:2.0:1789760791.753009:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25591808->25595903] (req 25591808->25595903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.753027:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25591808->25595903] (req 25591808->25595903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.753036:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25591808->25595903] (req 25591808->25595903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.753044:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760791.753049:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25591808->25595903] (req 25591808->25595903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.753059:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25591808->25595903] (req 25591808->25595903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.753101:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25591808->25595903] (req 25591808->25595903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760791.753108:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a495 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25591808->25595903] (req 25591808->25595903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760791.753198:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484370944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62548us (63352us total) trans 4294973566 rc 0/0 00000100:00100000:2.0:1789760791.753209:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6277 00000100:00100000:2.0:1789760791.753229:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780944980318 time 781021566530 state 0x2 ref 0 00000100:00100000:2.0:1789760791.764672:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760791.764851:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484371072 00000100:00100000:0.0:1789760791.764861:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484371072 00000100:00100000:0.0:1789760791.764920:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 780944980318 time 781021566530 state 0x0 ref 1 00000100:00100000:0.0:1789760791.764935:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781021826946 time 781021566530 state 0x1 ref 1 00000100:00100000:0.0:1789760791.764977:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781021826946 time 781095907630 state 0x0 ref 1 00000100:00100000:0.0:1789760791.764992:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484371072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760791.765045:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25595904, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:0.0:1789760791.765054:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.765101:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:0.0:1789760791.765106:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760791.765113:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25595904->25599999] (req 25595904->25599999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.765127:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25595904->25599999] (req 25595904->25599999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760791.814384:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973567 is committed 00002000:00010000:1.0:1789760791.814556:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25595904 -> 25600000 00010000:00010000:1.0:1789760791.814571:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25595904->25599999] (req 25595904->25599999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.814584:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25595904->25599999] (req 25595904->25599999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.814600:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25595904->25599999] (req 25595904->25599999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.814608:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.814612:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25595904->25599999] (req 25595904->25599999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.814622:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25595904->25599999] (req 25595904->25599999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.814641:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25595904->25599999] (req 25595904->25599999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.814649:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a49c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25595904->25599999] (req 25595904->25599999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.815604:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484371072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50608us (50932us total) trans 4294973567 rc 0/0 00000100:00100000:1.0:1789760791.815617:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6278 00000100:00100000:1.0:1789760791.815627:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781021826946 time 781095907630 state 0x2 ref 0 00000100:00100000:1.0:1789760791.823147:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760791.823430:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484371200 00000100:00100000:0.0:1789760791.823440:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484371200 00000100:00100000:0.0:1789760791.823508:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781021826946 time 781095907630 state 0x0 ref 1 00000100:00100000:0.0:1789760791.823523:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781096168046 time 781095907630 state 0x1 ref 1 00000100:00100000:0.0:1789760791.823553:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781096168046 time 781154484569 state 0x0 ref 1 00000100:00100000:0.0:1789760791.823567:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484371200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760791.823613:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25600000, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:0.0:1789760791.823622:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.823637:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:0.0:1789760791.823642:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760791.823649:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25600000->25604095] (req 25600000->25604095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760791.823663:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25600000->25604095] (req 25600000->25604095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760791.899180:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973568 is committed 00002000:00010000:3.0:1789760791.900151:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25600000 -> 25604096 00010000:00010000:3.0:1789760791.900170:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25600000->25604095] (req 25600000->25604095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.900183:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25600000->25604095] (req 25600000->25604095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.900192:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25600000->25604095] (req 25600000->25604095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.900201:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760791.900206:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25600000->25604095] (req 25600000->25604095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.900216:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25600000->25604095] (req 25600000->25604095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.900234:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25600000->25604095] (req 25600000->25604095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.900241:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a4a3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25600000->25604095] (req 25600000->25604095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760791.900442:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484371200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 76873us (77298us total) trans 4294973568 rc 0/0 00000100:00100000:0.0:1789760791.900454:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6279 00000100:00100000:0.0:1789760791.900464:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781096168046 time 781154484569 state 0x2 ref 0 00000100:00100000:0.0:1789760791.913510:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760791.913588:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484371328 00000100:00100000:1.0:1789760791.913599:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484371328 00000100:00100000:1.0:1789760791.913649:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781096168046 time 781154484569 state 0x0 ref 1 00000100:00100000:1.0:1789760791.913663:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781154744985 time 781154484569 state 0x1 ref 1 00000100:00100000:1.0:1789760791.913737:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781154744985 time 781244667592 state 0x0 ref 1 00000100:00100000:1.0:1789760791.913750:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484371328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760791.913801:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25604096, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:1.0:1789760791.913811:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.913826:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:1.0:1789760791.913831:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760791.913837:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25604096->25608191] (req 25604096->25608191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.913849:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25604096->25608191] (req 25604096->25608191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760791.972789:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25604096 -> 25608192 00010000:00010000:1.0:1789760791.972807:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25604096->25608191] (req 25604096->25608191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.972820:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25604096->25608191] (req 25604096->25608191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.972830:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25604096->25608191] (req 25604096->25608191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.972838:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760791.972844:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25604096->25608191] (req 25604096->25608191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.972853:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25604096->25608191] (req 25604096->25608191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.972871:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25604096->25608191] (req 25604096->25608191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760791.972878:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4aa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25604096->25608191] (req 25604096->25608191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760791.972977:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484371328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59224us (59474us total) trans 4294973569 rc 0/0 00000100:00100000:1.0:1789760791.972987:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6280 00000100:00100000:1.0:1789760791.972997:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781154744985 time 781244667592 state 0x2 ref 0 00000001:00080000:2.0:1789760791.973758:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973569 is committed 00000100:00100000:2.0:1789760791.977189:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760791.977412:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484371456 00000100:00100000:3.0:1789760791.977432:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484371456 00000100:00100000:3.0:1789760791.977488:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781154744985 time 781244667592 state 0x0 ref 1 00000100:00100000:3.0:1789760791.977502:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781244928008 time 781244667592 state 0x1 ref 1 00000100:00100000:3.0:1789760791.977537:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781244928008 time 781308468162 state 0x0 ref 1 00000100:00100000:3.0:1789760791.977549:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484371456:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760791.977595:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25608192, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:3.0:1789760791.977604:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.977621:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:3.0:1789760791.977625:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760791.977631:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25608192->25612287] (req 25608192->25612287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760791.977645:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25608192->25612287] (req 25608192->25612287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760792.013962:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973570 is committed 00002000:00010000:1.0:1789760792.014622:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25608192 -> 25612288 00010000:00010000:1.0:1789760792.014638:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25608192->25612287] (req 25608192->25612287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.014652:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25608192->25612287] (req 25608192->25612287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.014665:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25608192->25612287] (req 25608192->25612287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.014672:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760792.014676:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25608192->25612287] (req 25608192->25612287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.014684:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25608192->25612287] (req 25608192->25612287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.014702:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25608192->25612287] (req 25608192->25612287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.014710:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a4b1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25608192->25612287] (req 25608192->25612287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.014803:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484371456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37251us (37619us total) trans 4294973570 rc 0/0 00000100:00100000:1.0:1789760792.014813:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6281 00000100:00100000:1.0:1789760792.014822:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781244928008 time 781308468162 state 0x2 ref 0 00000100:00100000:3.0:1789760792.017994:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760792.018332:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484371584 00000100:00100000:0.0:1789760792.018343:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484371584 00000100:00100000:0.0:1789760792.018414:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781244928008 time 781308468162 state 0x0 ref 1 00000100:00100000:0.0:1789760792.018429:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781308728578 time 781308468162 state 0x1 ref 1 00000100:00080000:2.0:1789760792.018496:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1789760792.018539:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781308728578 time 781349469400 state 0x0 ref 1 00000100:00100000:0.0:1789760792.018560:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484371584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760792.018609:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25612288, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:0.0:1789760792.018618:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760792.018634:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:0.0:1789760792.018638:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760792.018644:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25612288->25616383] (req 25612288->25616383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760792.018658:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25612288->25616383] (req 25612288->25616383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760792.044772:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25612288 -> 25616384 00010000:00010000:1.0:1789760792.044792:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25612288->25616383] (req 25612288->25616383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.044816:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25612288->25616383] (req 25612288->25616383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.044825:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25612288->25616383] (req 25612288->25616383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.044831:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00000001:00080000:3.0:1789760792.044833:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973571 is committed 00010000:00010000:1.0:1789760792.044837:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25612288->25616383] (req 25612288->25616383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.044845:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25612288->25616383] (req 25612288->25616383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.044865:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25612288->25616383] (req 25612288->25616383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.044873:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a4b8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25612288->25616383] (req 25612288->25616383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.044975:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484371584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26417us (26985us total) trans 4294973571 rc 0/0 00000100:00100000:1.0:1789760792.044984:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6282 00000100:00100000:1.0:1789760792.044994:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781308728578 time 781349469400 state 0x2 ref 0 00000100:00100000:0.0:1789760792.048912:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760792.049184:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484371712 00000100:00100000:2.0:1789760792.049195:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484371712 00000100:00100000:2.0:1789760792.049250:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781308728578 time 781349469400 state 0x0 ref 1 00000100:00100000:2.0:1789760792.049264:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781349729816 time 781349469400 state 0x1 ref 1 00000100:00100000:2.0:1789760792.049324:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781349729816 time 781380255078 state 0x0 ref 1 00000100:00100000:2.0:1789760792.049338:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484371712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760792.049386:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25616384, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:2.0:1789760792.049397:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.049415:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:2.0:1789760792.049420:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760792.049427:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25616384->25620479] (req 25616384->25620479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.049441:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25616384->25620479] (req 25616384->25620479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760792.114704:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973572 is committed 00002000:00010000:1.0:1789760792.114925:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25616384 -> 25620480 00010000:00010000:1.0:1789760792.114952:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25616384->25620479] (req 25616384->25620479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.114965:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25616384->25620479] (req 25616384->25620479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.114974:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25616384->25620479] (req 25616384->25620479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.114981:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760792.114986:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25616384->25620479] (req 25616384->25620479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.114995:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25616384->25620479] (req 25616384->25620479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.115015:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25616384->25620479] (req 25616384->25620479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.115035:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a4bf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25616384->25620479] (req 25616384->25620479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.115199:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484371712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65860us (66292us total) trans 4294973572 rc 0/0 00000100:00100000:1.0:1789760792.115210:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6283 00000100:00100000:1.0:1789760792.115219:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781349729816 time 781380255078 state 0x2 ref 0 00000100:00100000:2.0:1789760792.118739:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760792.118860:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484371840 00000100:00100000:3.0:1789760792.118869:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484371840 00000100:00100000:3.0:1789760792.118921:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781349729816 time 781380255078 state 0x0 ref 1 00000100:00100000:3.0:1789760792.118946:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781380515494 time 781380255078 state 0x1 ref 1 00000100:00100000:3.0:1789760792.118977:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781380515494 time 781449907606 state 0x0 ref 1 00000100:00100000:3.0:1789760792.118989:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484371840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760792.119045:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25620480, mtime: 0x6aad9517, atime: 0x0, ctime: 0x6aad9517, blocks: 0xc807 00010000:00010000:3.0:1789760792.119055:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.119092:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:3.0:1789760792.119097:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760792.119102:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25620480->25624575] (req 25620480->25624575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.119117:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25620480->25624575] (req 25620480->25624575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760792.150817:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973573 is committed 00002000:00010000:3.0:1789760792.150919:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25620480 -> 25624576 00002000:00010000:3.0:1789760792.150924:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760791 -> 1789760792 00002000:00010000:3.0:1789760792.150927:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760791 -> 1789760792 00010000:00010000:3.0:1789760792.150952:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25620480->25624575] (req 25620480->25624575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.150966:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25620480->25624575] (req 25620480->25624575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.150975:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25620480->25624575] (req 25620480->25624575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.150983:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760792.150988:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25620480->25624575] (req 25620480->25624575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.150998:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25620480->25624575] (req 25620480->25624575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.151018:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25620480->25624575] (req 25620480->25624575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.151118:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4c6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25620480->25624575] (req 25620480->25624575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760792.151207:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484371840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32215us (32485us total) trans 4294973573 rc 0/0 00000100:00100000:3.0:1789760792.151216:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6284 00000100:00100000:3.0:1789760792.151226:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781380515494 time 781449907606 state 0x2 ref 0 00000100:00100000:2.0:1789760792.154992:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760792.155203:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484371968 00000100:00100000:1.0:1789760792.155215:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484371968 00000100:00100000:1.0:1789760792.155271:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781380515494 time 781449907606 state 0x0 ref 1 00000100:00100000:1.0:1789760792.155285:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781450168022 time 781449907606 state 0x1 ref 1 00000100:00100000:1.0:1789760792.155315:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781450168022 time 781486245826 state 0x0 ref 1 00000100:00100000:1.0:1789760792.155329:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484371968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760792.155376:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25624576, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:1.0:1789760792.155385:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.155401:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:1.0:1789760792.155405:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760792.155411:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25624576->25628671] (req 25624576->25628671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.155424:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25624576->25628671] (req 25624576->25628671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760792.185417:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25624576 -> 25628672 00010000:00010000:1.0:1789760792.185438:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25624576->25628671] (req 25624576->25628671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.185458:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25624576->25628671] (req 25624576->25628671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.185466:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25624576->25628671] (req 25624576->25628671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.185473:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760792.185478:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25624576->25628671] (req 25624576->25628671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.185486:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25624576->25628671] (req 25624576->25628671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.185506:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25624576->25628671] (req 25624576->25628671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.185513:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4cd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25624576->25628671] (req 25624576->25628671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.185615:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484371968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30284us (30625us total) trans 4294973574 rc 0/0 00000100:00100000:1.0:1789760792.185625:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6285 00000100:00100000:1.0:1789760792.185635:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781450168022 time 781486245826 state 0x2 ref 0 00000001:00080000:1.0:1789760792.185672:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973574 is committed 00000100:00100000:1.0:1789760792.188823:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760792.188924:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484372096 00000100:00100000:0.0:1789760792.188934:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484372096 00000100:00100000:0.0:1789760792.189011:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781450168022 time 781486245826 state 0x0 ref 1 00000100:00100000:0.0:1789760792.189027:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781486506242 time 781486245826 state 0x1 ref 1 00000100:00100000:0.0:1789760792.189374:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781486506242 time 781520302434 state 0x0 ref 1 00000100:00100000:0.0:1789760792.189390:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b18000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484372096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760792.189804:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25628672, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760792.189814:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.189831:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:2.0:1789760792.189835:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760792.189841:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25628672->25632767] (req 25628672->25632767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.189855:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25628672->25632767] (req 25628672->25632767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760792.226480:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25628672 -> 25632768 00010000:00010000:2.0:1789760792.226501:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25628672->25632767] (req 25628672->25632767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.226513:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25628672->25632767] (req 25628672->25632767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.226522:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25628672->25632767] (req 25628672->25632767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.226530:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.226534:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25628672->25632767] (req 25628672->25632767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.226543:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25628672->25632767] (req 25628672->25632767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.226566:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25628672->25632767] (req 25628672->25632767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.226574:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a4d4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25628672->25632767] (req 25628672->25632767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.226665:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484372096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37275us (37846us total) trans 4294973575 rc 0/0 00000100:00100000:2.0:1789760792.226676:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6286 00000100:00100000:2.0:1789760792.226685:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781486506242 time 781520302434 state 0x2 ref 0 00000001:00080000:1.0:1789760792.227590:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973575 is committed 00000100:00100000:2.0:1789760792.231910:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760792.232426:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484372224 00000100:00100000:1.0:1789760792.232437:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484372224 00000100:00100000:1.0:1789760792.232490:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781486506242 time 781520302434 state 0x0 ref 1 00000100:00100000:1.0:1789760792.232504:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781520562850 time 781520302434 state 0x1 ref 1 00000100:00100000:1.0:1789760792.232571:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781520562850 time 781563502024 state 0x0 ref 1 00000100:00100000:1.0:1789760792.232585:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760792.232634:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25632768, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:1.0:1789760792.232645:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.232660:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:1.0:1789760792.232664:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760792.232671:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25632768->25636863] (req 25632768->25636863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.232683:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25632768->25636863] (req 25632768->25636863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760792.270086:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973576 is committed 00002000:00010000:1.0:1789760792.270262:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25632768 -> 25636864 00010000:00010000:1.0:1789760792.270277:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25632768->25636863] (req 25632768->25636863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.270296:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25632768->25636863] (req 25632768->25636863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.270306:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25632768->25636863] (req 25632768->25636863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.270377:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760792.270383:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25632768->25636863] (req 25632768->25636863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.270395:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25632768->25636863] (req 25632768->25636863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.270413:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25632768->25636863] (req 25632768->25636863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.270421:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4db lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25632768->25636863] (req 25632768->25636863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.270541:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37954us (38636us total) trans 4294973576 rc 0/0 00000100:00100000:1.0:1789760792.270551:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6287 00000100:00100000:1.0:1789760792.270560:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781520562850 time 781563502024 state 0x2 ref 0 00000100:00100000:3.0:1789760792.277833:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760792.277893:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484372352 00000100:00100000:1.0:1789760792.277903:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484372352 00000100:00100000:1.0:1789760792.277979:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781520562850 time 781563502024 state 0x0 ref 1 00000100:00100000:1.0:1789760792.277995:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781563762440 time 781563502024 state 0x1 ref 1 00000100:00100000:1.0:1789760792.278025:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781563762440 time 781608956140 state 0x0 ref 1 00000100:00100000:1.0:1789760792.278038:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760792.278113:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25636864, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:1.0:1789760792.278122:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.278139:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:1.0:1789760792.278143:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760792.278150:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25636864->25640959] (req 25636864->25640959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.278163:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25636864->25640959] (req 25636864->25640959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760792.321651:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973577 is committed 00002000:00010000:1.0:1789760792.321849:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25636864 -> 25640960 00010000:00010000:1.0:1789760792.321864:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25636864->25640959] (req 25636864->25640959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.321875:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25636864->25640959] (req 25636864->25640959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.321888:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25636864->25640959] (req 25636864->25640959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.321895:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760792.321900:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25636864->25640959] (req 25636864->25640959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.321908:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25636864->25640959] (req 25636864->25640959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.321927:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25636864->25640959] (req 25636864->25640959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.321934:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4e2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25636864->25640959] (req 25636864->25640959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.322116:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886ea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44075us (44287us total) trans 4294973577 rc 0/0 00000100:00100000:1.0:1789760792.322127:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6288 00000100:00100000:1.0:1789760792.322137:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781563762440 time 781608956140 state 0x2 ref 0 00000100:00100000:0.0:1789760792.327889:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760792.327986:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484372480 00000100:00100000:3.0:1789760792.327997:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484372480 00000100:00100000:3.0:1789760792.328150:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781563762440 time 781608956140 state 0x0 ref 1 00000100:00100000:3.0:1789760792.328165:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781609216556 time 781608956140 state 0x1 ref 1 00000100:00100000:3.0:1789760792.328198:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781609216556 time 781659128682 state 0x0 ref 1 00000100:00100000:2.0:1789760792.328522:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760792.328570:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25640960, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760792.328578:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.328595:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:2.0:1789760792.328599:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760792.328605:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25640960->25645055] (req 25640960->25645055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.328619:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25640960->25645055] (req 25640960->25645055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760792.362414:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973578 is committed 00002000:00010000:1.0:1789760792.362678:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25640960 -> 25645056 00010000:00010000:1.0:1789760792.362693:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25640960->25645055] (req 25640960->25645055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.362711:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25640960->25645055] (req 25640960->25645055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.362838:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25640960->25645055] (req 25640960->25645055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.362848:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760792.362853:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25640960->25645055] (req 25640960->25645055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.362862:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25640960->25645055] (req 25640960->25645055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.362883:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25640960->25645055] (req 25640960->25645055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.362891:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a4e9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25640960->25645055] (req 25640960->25645055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.363038:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34609us (35321us total) trans 4294973578 rc 0/0 00000100:00100000:1.0:1789760792.363055:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6289 00000100:00100000:1.0:1789760792.363238:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781609216556 time 781659128682 state 0x2 ref 0 00000100:00100000:2.0:1789760792.372441:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760792.373201:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484372608 00000100:00100000:1.0:1789760792.373213:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484372608 00000100:00100000:1.0:1789760792.373274:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781609216556 time 781659128682 state 0x0 ref 1 00000100:00100000:1.0:1789760792.373290:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781659389098 time 781659128682 state 0x1 ref 1 00000100:00100000:1.0:1789760792.373325:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781659389098 time 781704255761 state 0x0 ref 1 00000100:00100000:1.0:1789760792.373338:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760792.373387:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25645056, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:1.0:1789760792.373396:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.373413:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:1.0:1789760792.373418:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760792.373424:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25645056->25649151] (req 25645056->25649151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.373438:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25645056->25649151] (req 25645056->25649151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760792.418668:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973579 is committed 00002000:00010000:1.0:1789760792.418880:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25645056 -> 25649152 00010000:00010000:1.0:1789760792.418897:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25645056->25649151] (req 25645056->25649151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.418910:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25645056->25649151] (req 25645056->25649151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.418925:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25645056->25649151] (req 25645056->25649151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.418944:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760792.418949:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25645056->25649151] (req 25645056->25649151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.418959:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25645056->25649151] (req 25645056->25649151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.418980:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25645056->25649151] (req 25645056->25649151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.418988:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25645056->25649151] (req 25645056->25649151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.419326:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45976us (46880us total) trans 4294973579 rc 0/0 00000100:00100000:1.0:1789760792.419337:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6290 00000100:00100000:1.0:1789760792.419347:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781659389098 time 781704255761 state 0x2 ref 0 00000100:00100000:3.0:1789760792.424991:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760792.425349:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484372736 00000100:00100000:1.0:1789760792.425359:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484372736 00000100:00100000:1.0:1789760792.425412:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781659389098 time 781704255761 state 0x0 ref 1 00000100:00100000:1.0:1789760792.425426:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781704516177 time 781704255761 state 0x1 ref 1 00000100:00100000:1.0:1789760792.425462:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781704516177 time 781756393595 state 0x0 ref 1 00000100:00100000:1.0:1789760792.425475:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484372736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760792.425524:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25649152, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:1.0:1789760792.425532:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.425548:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:1.0:1789760792.425553:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760792.425559:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25649152->25653247] (req 25649152->25653247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.425575:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25649152->25653247] (req 25649152->25653247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760792.451180:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973580 is committed 00002000:00010000:1.0:1789760792.451360:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25649152 -> 25653248 00010000:00010000:1.0:1789760792.451376:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25649152->25653247] (req 25649152->25653247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.451388:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25649152->25653247] (req 25649152->25653247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.451397:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25649152->25653247] (req 25649152->25653247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.451405:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760792.451411:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25649152->25653247] (req 25649152->25653247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.451419:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25649152->25653247] (req 25649152->25653247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.451438:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25649152->25653247] (req 25649152->25653247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.451447:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a4f7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25649152->25653247] (req 25649152->25653247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.451543:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484372736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26066us (26557us total) trans 4294973580 rc 0/0 00000100:00100000:1.0:1789760792.451553:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6291 00000100:00100000:1.0:1789760792.451563:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781704516177 time 781756393595 state 0x2 ref 0 00000100:00100000:2.0:1789760792.456506:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760792.456591:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484372864 00000100:00100000:3.0:1789760792.456602:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484372864 00000100:00100000:3.0:1789760792.456657:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781704516177 time 781756393595 state 0x0 ref 1 00000100:00100000:3.0:1789760792.456672:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781756654011 time 781756393595 state 0x1 ref 1 00000100:00100000:3.0:1789760792.456699:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781756654011 time 781787630555 state 0x0 ref 1 00000100:00100000:3.0:1789760792.456712:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760792.456763:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25653248, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:3.0:1789760792.456774:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.456790:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:3.0:1789760792.456795:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760792.456801:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25653248->25657343] (req 25653248->25657343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.456816:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25653248->25657343] (req 25653248->25657343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.461292:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760792.484475:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973581 is committed 00002000:00010000:1.0:1789760792.485248:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25653248 -> 25657344 00010000:00010000:1.0:1789760792.485266:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25653248->25657343] (req 25653248->25657343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.485279:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25653248->25657343] (req 25653248->25657343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.485288:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25653248->25657343] (req 25653248->25657343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.485301:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760792.485307:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25653248->25657343] (req 25653248->25657343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.485316:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25653248->25657343] (req 25653248->25657343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.485338:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25653248->25657343] (req 25653248->25657343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.485345:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a4fe lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25653248->25657343] (req 25653248->25657343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760792.485448:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28734us (28947us total) trans 4294973581 rc 0/0 00000100:00100000:1.0:1789760792.485459:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6292 00000100:00100000:1.0:1789760792.485467:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781756654011 time 781787630555 state 0x2 ref 0 00000100:00100000:3.0:1789760792.489600:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760792.489734:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484372992 00000100:00100000:2.0:1789760792.489745:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484372992 00000100:00100000:2.0:1789760792.489796:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781756654011 time 781787630555 state 0x0 ref 1 00000100:00100000:2.0:1789760792.489811:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781787890971 time 781787630555 state 0x1 ref 1 00000100:00100000:2.0:1789760792.489848:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781787890971 time 781820779141 state 0x0 ref 1 00000100:00100000:2.0:1789760792.489863:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484372992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760792.489913:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25657344, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760792.489923:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.489950:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:2.0:1789760792.489954:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760792.489962:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25657344->25661439] (req 25657344->25661439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.489976:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25657344->25661439] (req 25657344->25661439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760792.536411:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25657344 -> 25661440 00010000:00010000:2.0:1789760792.536431:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25657344->25661439] (req 25657344->25661439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.536445:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25657344->25661439] (req 25657344->25661439) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.536455:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25657344->25661439] (req 25657344->25661439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.536463:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.536467:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25657344->25661439] (req 25657344->25661439) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.536477:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25657344->25661439] (req 25657344->25661439) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.536495:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25657344->25661439] (req 25657344->25661439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.536503:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a505 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25657344->25661439] (req 25657344->25661439) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.536599:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c886e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484372992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46734us (47003us total) trans 4294973582 rc 0/0 00000100:00100000:2.0:1789760792.536614:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6293 00000100:00100000:2.0:1789760792.536623:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781787890971 time 781820779141 state 0x2 ref 0 00000001:00080000:1.0:1789760792.536697:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973582 is committed 00000100:00100000:3.0:1789760792.543765:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760792.544038:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484373120 00000100:00100000:1.0:1789760792.544048:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484373120 00000100:00100000:1.0:1789760792.544378:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781787890971 time 781820779141 state 0x0 ref 1 00000100:00100000:1.0:1789760792.544395:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781821039557 time 781820779141 state 0x1 ref 1 00000100:00100000:1.0:1789760792.544470:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781821039557 time 781875400096 state 0x0 ref 1 00000100:00100000:1.0:1789760792.544485:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484373120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760792.544533:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25661440, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:1.0:1789760792.544543:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.544559:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:1.0:1789760792.544563:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760792.544569:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25661440->25665535] (req 25661440->25665535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.544583:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25661440->25665535] (req 25661440->25665535) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760792.589566:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973583 is committed 00002000:00010000:2.0:1789760792.589677:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25661440 -> 25665536 00010000:00010000:2.0:1789760792.589693:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25661440->25665535] (req 25661440->25665535) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.589707:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25661440->25665535] (req 25661440->25665535) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.589726:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25661440->25665535] (req 25661440->25665535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.589734:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.589739:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25661440->25665535] (req 25661440->25665535) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.589748:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25661440->25665535] (req 25661440->25665535) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.589767:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25661440->25665535] (req 25661440->25665535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.589774:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a50c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25661440->25665535] (req 25661440->25665535) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.589878:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484373120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45391us (46119us total) trans 4294973583 rc 0/0 00000100:00100000:2.0:1789760792.589889:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6294 00000100:00100000:2.0:1789760792.589899:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781821039557 time 781875400096 state 0x2 ref 0 00000100:00100000:3.0:1789760792.598392:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760792.598579:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484373248 00000100:00100000:1.0:1789760792.598589:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484373248 00000100:00100000:1.0:1789760792.598640:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781821039557 time 781875400096 state 0x0 ref 1 00000100:00100000:1.0:1789760792.598655:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781875660512 time 781875400096 state 0x1 ref 1 00000100:00100000:1.0:1789760792.598684:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781875660512 time 781929615061 state 0x0 ref 1 00000100:00100000:1.0:1789760792.598695:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7069c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484373248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760792.598750:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25665536, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:1.0:1789760792.598759:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.598775:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:1.0:1789760792.598780:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760792.598787:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25665536->25669631] (req 25665536->25669631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.598800:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25665536->25669631] (req 25665536->25669631) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760792.630760:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25665536 -> 25669632 00010000:00010000:2.0:1789760792.630778:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25665536->25669631] (req 25665536->25669631) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.630792:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25665536->25669631] (req 25665536->25669631) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.630803:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25665536->25669631] (req 25665536->25669631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.630810:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.630816:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25665536->25669631] (req 25665536->25669631) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.630834:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25665536->25669631] (req 25665536->25669631) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.630854:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25665536->25669631] (req 25665536->25669631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.630867:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a513 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25665536->25669631] (req 25665536->25669631) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.630966:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7069c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484373248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32265us (32577us total) trans 4294973584 rc 0/0 00000100:00100000:2.0:1789760792.630978:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6295 00000100:00100000:2.0:1789760792.630988:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781875660512 time 781929615061 state 0x2 ref 0 00000001:00080000:3.0:1789760792.631966:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973584 is committed 00000100:00100000:1.0:1789760792.634755:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760792.634828:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484373376 00000100:00100000:3.0:1789760792.634837:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484373376 00000100:00100000:3.0:1789760792.634892:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781875660512 time 781929615061 state 0x0 ref 1 00000100:00100000:3.0:1789760792.634907:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781929875477 time 781929615061 state 0x1 ref 1 00000100:00100000:3.0:1789760792.634982:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781929875477 time 781965911929 state 0x0 ref 1 00000100:00100000:3.0:1789760792.634996:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484373376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760792.635041:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25669632, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:3.0:1789760792.635050:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.635087:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:3.0:1789760792.635091:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760792.635098:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25669632->25673727] (req 25669632->25673727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.635113:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25669632->25673727] (req 25669632->25673727) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760792.664255:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973585 is committed 00002000:00010000:0.0:1789760792.664357:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25669632 -> 25673728 00010000:00010000:0.0:1789760792.664372:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25669632->25673727] (req 25669632->25673727) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760792.664391:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25669632->25673727] (req 25669632->25673727) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760792.664399:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25669632->25673727] (req 25669632->25673727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760792.664406:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760792.664411:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25669632->25673727] (req 25669632->25673727) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760792.664421:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25669632->25673727] (req 25669632->25673727) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760792.664440:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25669632->25673727] (req 25669632->25673727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760792.664448:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a51a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25669632->25673727] (req 25669632->25673727) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760792.664541:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484373376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29544us (29790us total) trans 4294973585 rc 0/0 00000100:00100000:0.0:1789760792.664551:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6296 00000100:00100000:0.0:1789760792.664561:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781929875477 time 781965911929 state 0x2 ref 0 00000100:00100000:0.0:1789760792.669621:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760792.669703:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484373504 00000100:00100000:2.0:1789760792.669712:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484373504 00000100:00100000:2.0:1789760792.669770:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781929875477 time 781965911929 state 0x0 ref 1 00000100:00100000:2.0:1789760792.669786:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781966172345 time 781965911929 state 0x1 ref 1 00000100:00100000:2.0:1789760792.669820:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781966172345 time 782000750872 state 0x0 ref 1 00000100:00100000:2.0:1789760792.669832:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484373504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760792.669881:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25673728, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760792.669891:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.669907:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=781s v=(0 0) 00010000:00010000:2.0:1789760792.669911:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760792.669929:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25673728->25677823] (req 25673728->25677823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.669952:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25673728->25677823] (req 25673728->25677823) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760792.701847:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973586 is committed 00002000:00010000:3.0:1789760792.702022:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25673728 -> 25677824 00010000:00010000:3.0:1789760792.702037:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25673728->25677823] (req 25673728->25677823) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.702050:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25673728->25677823] (req 25673728->25677823) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.702059:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25673728->25677823] (req 25673728->25677823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.702098:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760792.702103:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25673728->25677823] (req 25673728->25677823) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.702112:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25673728->25677823] (req 25673728->25677823) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.702135:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25673728->25677823] (req 25673728->25677823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.702142:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a521 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25673728->25677823] (req 25673728->25677823) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760792.702225:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484373504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32391us (32611us total) trans 4294973586 rc 0/0 00000100:00100000:3.0:1789760792.702235:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6297 00000100:00100000:3.0:1789760792.702245:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781966172345 time 782000750872 state 0x2 ref 0 00000100:00100000:3.0:1789760792.706396:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760792.706497:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484373632 00000100:00100000:2.0:1789760792.706508:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484373632 00000100:00100000:2.0:1789760792.706563:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 781966172345 time 782000750872 state 0x0 ref 1 00000100:00100000:2.0:1789760792.706580:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782001011288 time 782000750872 state 0x1 ref 1 00000100:00100000:2.0:1789760792.706613:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782001011288 time 782037543819 state 0x0 ref 1 00000100:00100000:2.0:1789760792.706627:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9652300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484373632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760792.706675:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25677824, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760792.706684:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.706700:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:2.0:1789760792.706705:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760792.706712:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25677824->25681919] (req 25677824->25681919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.706733:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25677824->25681919] (req 25677824->25681919) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760792.736939:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973587 is committed 00002000:00010000:2.0:1789760792.737152:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25677824 -> 25681920 00010000:00010000:2.0:1789760792.737169:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25677824->25681919] (req 25677824->25681919) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.737183:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25677824->25681919] (req 25677824->25681919) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.737192:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25677824->25681919] (req 25677824->25681919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.737201:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.737206:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25677824->25681919] (req 25677824->25681919) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.737216:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25677824->25681919] (req 25677824->25681919) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.737236:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25677824->25681919] (req 25677824->25681919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.737244:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a528 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25677824->25681919] (req 25677824->25681919) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.737415:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9652300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484373632:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30785us (31024us total) trans 4294973587 rc 0/0 00000100:00100000:2.0:1789760792.737426:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6298 00000100:00100000:2.0:1789760792.737436:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782001011288 time 782037543819 state 0x2 ref 0 00000100:00100000:0.0:1789760792.741194:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760792.741275:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484373760 00000100:00100000:1.0:1789760792.741286:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484373760 00000100:00100000:1.0:1789760792.741337:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782001011288 time 782037543819 state 0x0 ref 1 00000100:00100000:1.0:1789760792.741353:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782037804235 time 782037543819 state 0x1 ref 1 00000100:00100000:1.0:1789760792.741382:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782037804235 time 782072313281 state 0x0 ref 1 00000100:00100000:1.0:1789760792.741394:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484373760:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760792.741441:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25681920, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:1.0:1789760792.741451:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.741467:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:1.0:1789760792.741471:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760792.741478:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25681920->25686015] (req 25681920->25686015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760792.741493:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25681920->25686015] (req 25681920->25686015) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760792.774666:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973588 is committed 00002000:00010000:2.0:1789760792.774849:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25681920 -> 25686016 00010000:00010000:2.0:1789760792.774864:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25681920->25686015] (req 25681920->25686015) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.774878:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25681920->25686015] (req 25681920->25686015) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.774888:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25681920->25686015] (req 25681920->25686015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.774896:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.774901:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25681920->25686015] (req 25681920->25686015) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.774911:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25681920->25686015] (req 25681920->25686015) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.774940:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25681920->25686015] (req 25681920->25686015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.774949:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a52f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25681920->25686015] (req 25681920->25686015) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.775156:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484373760:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33760us (33967us total) trans 4294973588 rc 0/0 00000100:00100000:2.0:1789760792.775166:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6299 00000100:00100000:2.0:1789760792.775177:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782037804235 time 782072313281 state 0x2 ref 0 00000100:00100000:0.0:1789760792.778649:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760792.778773:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484373888 00000100:00100000:3.0:1789760792.778784:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484373888 00000100:00100000:3.0:1789760792.778837:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782037804235 time 782072313281 state 0x0 ref 1 00000100:00100000:3.0:1789760792.778852:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782072573697 time 782072313281 state 0x1 ref 1 00000100:00100000:3.0:1789760792.778921:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782072573697 time 782109851434 state 0x0 ref 1 00000100:00100000:3.0:1789760792.778944:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484373888:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760792.778993:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25686016, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:3.0:1789760792.779002:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.779019:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:3.0:1789760792.779024:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760792.779030:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25686016->25690111] (req 25686016->25690111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.779045:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25686016->25690111] (req 25686016->25690111) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760792.802839:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973589 is committed 00002000:00010000:2.0:1789760792.803189:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25686016 -> 25690112 00010000:00010000:2.0:1789760792.803206:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25686016->25690111] (req 25686016->25690111) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.803219:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25686016->25690111] (req 25686016->25690111) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.803227:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25686016->25690111] (req 25686016->25690111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.803235:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.803240:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25686016->25690111] (req 25686016->25690111) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.803249:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25686016->25690111] (req 25686016->25690111) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.803269:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25686016->25690111] (req 25686016->25690111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.803277:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a536 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25686016->25690111] (req 25686016->25690111) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.803365:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484373888:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24428us (24721us total) trans 4294973589 rc 0/0 00000100:00100000:2.0:1789760792.803380:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6300 00000100:00100000:2.0:1789760792.803390:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782072573697 time 782109851434 state 0x2 ref 0 00000100:00100000:3.0:1789760792.808313:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760792.808414:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484374016 00000100:00100000:2.0:1789760792.808425:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484374016 00000100:00100000:2.0:1789760792.808478:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782072573697 time 782109851434 state 0x0 ref 1 00000100:00100000:2.0:1789760792.808494:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782110111850 time 782109851434 state 0x1 ref 1 00000100:00100000:2.0:1789760792.808530:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782110111850 time 782139461111 state 0x0 ref 1 00000100:00100000:2.0:1789760792.808542:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9651c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484374016:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760792.808589:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25690112, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760792.808597:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.808623:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:2.0:1789760792.808628:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760792.808634:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25690112->25694207] (req 25690112->25694207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.808649:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25690112->25694207] (req 25690112->25694207) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760792.846482:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25690112 -> 25694208 00010000:00010000:2.0:1789760792.846504:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25690112->25694207] (req 25690112->25694207) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.846518:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25690112->25694207] (req 25690112->25694207) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.846528:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25690112->25694207] (req 25690112->25694207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.846535:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.846540:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25690112->25694207] (req 25690112->25694207) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.846550:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25690112->25694207] (req 25690112->25694207) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.846573:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25690112->25694207] (req 25690112->25694207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.846581:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a53d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25690112->25694207] (req 25690112->25694207) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.846672:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9651c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484374016:12345-192.168.203.10@tcp:4:dd.0 Request processed in 38126us (38363us total) trans 4294973590 rc 0/0 00000100:00100000:2.0:1789760792.846683:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6301 00000100:00100000:2.0:1789760792.846693:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782110111850 time 782139461111 state 0x2 ref 0 00000001:00080000:2.0:1789760792.848138:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973590 is committed 00000100:00100000:0.0:1789760792.851634:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760792.851719:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484374144 00000100:00100000:2.0:1789760792.851729:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484374144 00000100:00100000:2.0:1789760792.851781:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782110111850 time 782139461111 state 0x0 ref 1 00000100:00100000:2.0:1789760792.851797:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782139721527 time 782139461111 state 0x1 ref 1 00000100:00100000:2.0:1789760792.851827:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782139721527 time 782182758074 state 0x0 ref 1 00000100:00100000:2.0:1789760792.851839:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484374144:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760792.851899:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25694208, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760792.851908:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.851925:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:2.0:1789760792.851946:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760792.851953:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25694208->25698303] (req 25694208->25698303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.851969:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25694208->25698303] (req 25694208->25698303) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760792.880752:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25694208 -> 25698304 00010000:00010000:2.0:1789760792.880774:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25694208->25698303] (req 25694208->25698303) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.880786:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25694208->25698303] (req 25694208->25698303) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.880800:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25694208->25698303] (req 25694208->25698303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.880808:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.880813:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25694208->25698303] (req 25694208->25698303) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.880820:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25694208->25698303] (req 25694208->25698303) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.880838:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25694208->25698303] (req 25694208->25698303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.880845:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a544 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25694208->25698303] (req 25694208->25698303) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.881129:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484374144:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29285us (29498us total) trans 4294973591 rc 0/0 00000100:00100000:2.0:1789760792.881139:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6302 00000100:00100000:2.0:1789760792.881149:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782139721527 time 782182758074 state 0x2 ref 0 00000001:00080000:2.0:1789760792.881187:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973591 is committed 00000100:00100000:1.0:1789760792.885687:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760792.885846:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484374272 00000100:00100000:2.0:1789760792.885866:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484374272 00000100:00100000:2.0:1789760792.885921:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782139721527 time 782182758074 state 0x0 ref 1 00000100:00100000:2.0:1789760792.885951:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782183018490 time 782182758074 state 0x1 ref 1 00000100:00100000:2.0:1789760792.885975:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782183018490 time 782216905957 state 0x0 ref 1 00000100:00100000:2.0:1789760792.885987:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484374272:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760792.886036:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25698304, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760792.886045:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.886100:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:2.0:1789760792.886105:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760792.886113:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25698304->25702399] (req 25698304->25702399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.886128:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25698304->25702399] (req 25698304->25702399) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760792.919452:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25698304 -> 25702400 00010000:00010000:3.0:1789760792.919471:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25698304->25702399] (req 25698304->25702399) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.919483:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25698304->25702399] (req 25698304->25702399) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.919492:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25698304->25702399] (req 25698304->25702399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.919498:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760792.919504:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25698304->25702399] (req 25698304->25702399) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.919512:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25698304->25702399] (req 25698304->25702399) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.919532:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25698304->25702399] (req 25698304->25702399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.919539:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a54b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25698304->25702399] (req 25698304->25702399) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760792.919625:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484374272:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33636us (33945us total) trans 4294973592 rc 0/0 00000100:00100000:3.0:1789760792.919635:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6303 00000100:00100000:3.0:1789760792.919644:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782183018490 time 782216905957 state 0x2 ref 0 00000001:00080000:0.0:1789760792.920340:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973592 is committed 00000100:00100000:2.0:1789760792.923573:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760792.923665:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484374400 00000100:00100000:3.0:1789760792.923675:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484374400 00000100:00100000:3.0:1789760792.923726:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782183018490 time 782216905957 state 0x0 ref 1 00000100:00100000:3.0:1789760792.923740:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782217166373 time 782216905957 state 0x1 ref 1 00000100:00100000:3.0:1789760792.923803:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782217166373 time 782254734649 state 0x0 ref 1 00000100:00100000:3.0:1789760792.923816:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484374400:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760792.923867:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25702400, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:3.0:1789760792.923876:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.923891:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:3.0:1789760792.923896:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760792.923902:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25702400->25706495] (req 25702400->25706495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.923924:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25702400->25706495] (req 25702400->25706495) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760792.960625:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973593 is committed 00002000:00010000:2.0:1789760792.960739:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25702400 -> 25706496 00010000:00010000:2.0:1789760792.960754:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25702400->25706495] (req 25702400->25706495) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.960767:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25702400->25706495] (req 25702400->25706495) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.960776:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25702400->25706495] (req 25702400->25706495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.960785:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760792.960790:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25702400->25706495] (req 25702400->25706495) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.960798:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25702400->25706495] (req 25702400->25706495) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.960816:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25702400->25706495] (req 25702400->25706495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760792.960824:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a552 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25702400->25706495] (req 25702400->25706495) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.960950:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb5500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484374400:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37132us (37382us total) trans 4294973593 rc 0/0 00000100:00100000:2.0:1789760792.960960:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6304 00000100:00100000:2.0:1789760792.960969:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782217166373 time 782254734649 state 0x2 ref 0 00000100:00100000:0.0:1789760792.965483:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760792.965561:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484374528 00000100:00100000:3.0:1789760792.965580:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484374528 00000100:00100000:3.0:1789760792.965636:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782217166373 time 782254734649 state 0x0 ref 1 00000100:00100000:3.0:1789760792.965651:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782254995065 time 782254734649 state 0x1 ref 1 00000100:00100000:3.0:1789760792.965680:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782254995065 time 782296611510 state 0x0 ref 1 00000100:00100000:3.0:1789760792.965692:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484374528:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760792.965740:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25706496, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:3.0:1789760792.965750:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.965768:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:3.0:1789760792.965773:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760792.965779:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25706496->25710591] (req 25706496->25710591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.965793:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25706496->25710591] (req 25706496->25710591) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760792.972301:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4dfbb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508943616:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760792.972385:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760792.972447:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9650a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508944128:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.972474:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760792.972500:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508944640:0@lo:400:kworker.0 00000100:00100000:3.0:1789760792.972504:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508943616 00000100:00100000:3.0:1789760792.972512:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508943616 00000100:00100000:2.0:1789760792.972523:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760792.972540:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508945152:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.972559:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760792.972563:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 420 00000100:00100000:2.0:1789760792.972576:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:2.0:1789760792.972581:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9651f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508943744:0@lo:400:kworker.0 00000100:00100000:3.0:1789760792.972582:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508943616:12345-0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760792.972604:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00002000:00000024:3.0:1789760792.972613:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760792.972622:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508944256:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.972642:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760792.972650:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508943616:12345-0@lo:13:osp-pre-1-0.0 Request processed in 79us (264us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760792.972656:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 420 00000100:00100000:2.0:1789760792.972657:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508944768:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.972674:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760792.972690:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8ab8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508945280:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.972708:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1789760792.972753:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4dfbb6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508943616:0@lo:13:osp-pre-1-0.0 00000100:00080000:2.0:1789760792.972781:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1789760792.972792:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508944640 00000100:00100000:2.0:1789760792.972795:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508944640 00000100:00100000:2.0:1789760792.972826:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508945152 00000100:00100000:3.0:1789760792.972827:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508944768 00000100:00100000:2.0:1789760792.972829:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508945152 00000100:00100000:3.0:1789760792.972829:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508944768 00000100:00100000:2.0:1789760792.972842:0:8307:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508945280 00000100:00100000:2.0:1789760792.972844:0:8307:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508945280 00000100:00100000:3.0:1789760792.972845:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 353 00000100:00100000:2.0:1789760792.972854:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 354 00000100:00100000:2.0:1789760792.972860:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508945152:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760792.972862:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508944640:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760792.972894:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb7800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508944640:12345-0@lo:400:kworker.0 Request processed in 43us (370us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760792.972896:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dc380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0001-mdtlov_UUID+5:3478:x1876699508945152:12345-0@lo:400:kworker.0 Request processed in 37us (337us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760792.972901:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 353 00000100:00100000:2.0:1789760792.972904:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 354 00000100:00100000:3.0:1789760792.972913:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 355 00000100:00100000:2.0:1789760792.972916:0:8307:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 356 00000100:00100000:3.0:1789760792.972918:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508944768:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.972921:0:8307:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508945280:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.972955:0:8307:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508945280:12345-0@lo:400:kworker.0 Request processed in 33us (245us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760792.972955:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508944768:12345-0@lo:400:kworker.0 Request processed in 36us (278us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760792.972961:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 355 00000100:00100000:2.0:1789760792.972964:0:8307:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 356 00000100:00100000:2.0:1789760792.972979:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508943744 00000100:00100000:2.0:1789760792.972982:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508943744 00000100:00100000:2.0:1789760792.973013:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 498 00000100:00100000:2.0:1789760792.973029:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8038c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508943744:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.973058:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8038c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508943744:12345-0@lo:400:kworker.0 Request processed in 39us (453us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760792.973104:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 498 00000100:00100000:2.0:1789760792.973130:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508944256 00000100:00100000:2.0:1789760792.973134:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508944256 00000100:00100000:2.0:1789760792.973160:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 479 00000100:00100000:2.0:1789760792.973167:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508944256:12345-0@lo:400:kworker.0 00000100:00080000:3.0:1789760792.973177:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:3.0:1789760792.973180:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1789760792.973197:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c98dea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508944256:12345-0@lo:400:kworker.0 Request processed in 30us (553us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760792.973203:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 479 00000100:00100000:2.0:1789760792.973226:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9651f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508943744:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.973236:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508944256:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.973244:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508944768:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.973253:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8ab8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508945280:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.973261:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508944640:0@lo:400:kworker.0 00000100:00100000:2.0:1789760792.973269:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0001-mdtlov_UUID:3478:1876699508945152:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.973534:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9653b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508943872:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.973567:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760792.973601:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508944384:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.973619:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760792.973634:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508944896:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.973651:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760792.973668:0:3480:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [3->2] 00000100:00100000:1.0:1789760792.973673:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9652d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508944000:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.973694:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760792.973708:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508944512:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.973733:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760792.973746:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9b1e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508945024:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.973766:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760792.973794:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508944128 00000100:00100000:1.0:1789760792.973799:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508944128 00000100:00100000:1.0:1789760792.973821:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508943872 00000100:00100000:1.0:1789760792.973823:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508943872 00000100:00100000:1.0:1789760792.973833:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508944384 00000100:00100000:1.0:1789760792.973835:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508944384 00000100:00100000:1.0:1789760792.973860:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508944896 00000100:00100000:1.0:1789760792.973863:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508944896 00000100:00100000:1.0:1789760792.973876:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508944512 00000100:00100000:1.0:1789760792.973878:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508944512 00000100:00100000:1.0:1789760792.973889:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508945024 00000100:00100000:1.0:1789760792.973892:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508945024 00000100:00100000:1.0:1789760792.973904:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1015 00000100:00100000:1.0:1789760792.973910:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508944128:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.973959:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb4000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0001_UUID+5:3478:x1876699508944128:12345-0@lo:400:kworker.0 Request processed in 49us (1484us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760792.973965:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1015 00000100:00100000:1.0:1789760792.973979:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1016 00000100:00100000:1.0:1789760792.973984:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508943872:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.974013:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3480:x1876699508943872:12345-0@lo:400:kworker.0 Request processed in 29us (446us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760792.974019:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1016 00000100:00100000:1.0:1789760792.974523:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1017 00000100:00100000:1.0:1789760792.974536:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508944384:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.974622:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0000_UUID+5:3480:x1876699508944384:12345-0@lo:400:kworker.0 Request processed in 38us (949us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760792.974628:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1017 00000100:00100000:1.0:1789760792.974641:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1018 00000100:00100000:1.0:1789760792.974646:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1bb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508944896:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.974670:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1bb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0001_UUID+5:3480:x1876699508944896:12345-0@lo:400:kworker.0 Request processed in 25us (1018us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760792.974675:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1018 00000100:00100000:1.0:1789760792.974685:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1019 00000100:00100000:1.0:1789760792.974691:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b18000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508944512:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.974919:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b18000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0000_UUID+5:3480:x1876699508944512:12345-0@lo:400:kworker.0 Request processed in 23us (980us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760792.974927:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1019 00000100:00100000:1.0:1789760792.974951:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1020 00000100:00100000:1.0:1789760792.974958:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508945024:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.974987:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f11500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0001-lwp-OST0001_UUID+5:3480:x1876699508945024:12345-0@lo:400:kworker.0 Request processed in 30us (1220us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760792.974993:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1020 00000100:00100000:1.0:1789760792.975539:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508944000 00000100:00100000:1.0:1789760792.975547:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508944000 00000100:00100000:1.0:1789760792.976218:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 480 00000100:00100000:1.0:1789760792.976229:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508944000:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.976286:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0001-mdtlov_UUID+6:3480:x1876699508944000:12345-0@lo:400:kworker.0 Request processed in 60us (2590us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760792.976292:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 480 00000100:00080000:2.0:1789760792.976338:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:0.0:1789760792.976938:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1789760792.976953:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c8ab9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508945408:0@lo:13:osp-pre-1-1.0 00000100:00100000:0.0:1789760792.976995:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760792.977152:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508945408 00000100:00100000:3.0:1789760792.977159:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508945408 00000100:00100000:3.0:1789760792.977187:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 421 00000100:00100000:3.0:1789760792.977194:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508945408:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:3.0:1789760792.977213:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:3.0:1789760792.977251:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5a680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508945408:12345-0@lo:13:osp-pre-1-1.0 Request processed in 57us (256us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760792.977257:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 421 00000100:00100000:3.0:1789760792.977294:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c8ab9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508945408:0@lo:13:osp-pre-1-1.0 00000100:00080000:2.0:1789760792.977307:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:2.0:1789760792.977331:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9650a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0001_UUID:3478:1876699508944128:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.979081:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9652d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-mdtlov_UUID:3480:1876699508944000:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.979098:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0000_UUID:3480:1876699508944512:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.979110:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0001-lwp-OST0001_UUID:3480:1876699508945024:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.979121:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9653b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-MDT0000_UUID:3480:1876699508943872:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.979133:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3480:1876699508944384:0@lo:400:kworker.0 00000100:00100000:1.0:1789760792.979144:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9b1d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3480:1876699508944896:0@lo:400:kworker.0 00002000:00010000:3.0:1789760792.994775:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25706496 -> 25710592 00010000:00010000:3.0:1789760792.994795:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25706496->25710591] (req 25706496->25710591) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.994808:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25706496->25710591] (req 25706496->25710591) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.994818:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25706496->25710591] (req 25706496->25710591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.994825:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760792.994830:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25706496->25710591] (req 25706496->25710591) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.994840:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25706496->25710591] (req 25706496->25710591) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.994859:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25706496->25710591] (req 25706496->25710591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760792.994868:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a559 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25706496->25710591] (req 25706496->25710591) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760792.994973:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484374528:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29276us (29492us total) trans 4294973594 rc 0/0 00000100:00100000:3.0:1789760792.994984:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6305 00000100:00100000:3.0:1789760792.994993:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782254995065 time 782296611510 state 0x2 ref 0 00000001:00080000:1.0:1789760792.995590:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973594 is committed 00000100:00100000:1.0:1789760793.001915:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760793.002108:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484374656 00000100:00100000:2.0:1789760793.002118:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484374656 00000100:00100000:2.0:1789760793.002171:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782254995065 time 782296611510 state 0x0 ref 1 00000100:00100000:2.0:1789760793.002186:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782296871926 time 782296611510 state 0x1 ref 1 00000100:00100000:2.0:1789760793.002209:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782296871926 time 782333140550 state 0x0 ref 1 00000100:00100000:2.0:1789760793.002221:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484374656:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760793.002268:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25710592, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760793.002276:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.002292:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:2.0:1789760793.002296:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760793.002302:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25710592->25714687] (req 25710592->25714687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.002317:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25710592->25714687] (req 25710592->25714687) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760793.033862:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973595 is committed 00002000:00010000:2.0:1789760793.034032:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25710592 -> 25714688 00010000:00010000:2.0:1789760793.034047:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25710592->25714687] (req 25710592->25714687) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.034060:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25710592->25714687] (req 25710592->25714687) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.034104:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25710592->25714687] (req 25710592->25714687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.034112:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760793.034117:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25710592->25714687] (req 25710592->25714687) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.034127:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25710592->25714687] (req 25710592->25714687) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.034153:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25710592->25714687] (req 25710592->25714687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.034161:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a560 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25710592->25714687] (req 25710592->25714687) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.034246:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484374656:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32021us (32334us total) trans 4294973595 rc 0/0 00000100:00100000:2.0:1789760793.034257:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6306 00000100:00100000:2.0:1789760793.034267:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782296871926 time 782333140550 state 0x2 ref 0 00000100:00100000:2.0:1789760793.037445:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760793.037526:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484374784 00000100:00100000:3.0:1789760793.037536:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484374784 00000100:00100000:3.0:1789760793.037590:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782296871926 time 782333140550 state 0x0 ref 1 00000100:00100000:3.0:1789760793.037604:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782333400966 time 782333140550 state 0x1 ref 1 00000100:00100000:3.0:1789760793.037633:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782333400966 time 782368564485 state 0x0 ref 1 00000100:00100000:3.0:1789760793.037646:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484374784:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760793.037693:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25714688, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:3.0:1789760793.037702:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.037726:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:3.0:1789760793.037731:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760793.037737:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25714688->25718783] (req 25714688->25718783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.037757:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25714688->25718783] (req 25714688->25718783) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760793.073545:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25714688 -> 25718784 00010000:00010000:0.0:1789760793.073563:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25714688->25718783] (req 25714688->25718783) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.073575:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25714688->25718783] (req 25714688->25718783) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.073583:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25714688->25718783] (req 25714688->25718783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.073589:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760793.073594:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25714688->25718783] (req 25714688->25718783) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.073603:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25714688->25718783] (req 25714688->25718783) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.073627:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25714688->25718783] (req 25714688->25718783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.073635:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a567 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25714688->25718783] (req 25714688->25718783) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760793.073730:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c46a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484374784:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36082us (36291us total) trans 4294973596 rc 0/0 00000100:00100000:0.0:1789760793.073741:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6307 00000100:00100000:0.0:1789760793.073751:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782333400966 time 782368564485 state 0x2 ref 0 00000001:00080000:1.0:1789760793.076762:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973596 is committed 00000100:00100000:0.0:1789760793.077757:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760793.077844:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484374912 00000100:00100000:2.0:1789760793.077854:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484374912 00000100:00100000:2.0:1789760793.077907:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782333400966 time 782368564485 state 0x0 ref 1 00000100:00100000:2.0:1789760793.077923:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782368824901 time 782368564485 state 0x1 ref 1 00000100:00100000:2.0:1789760793.077965:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782368824901 time 782408895884 state 0x0 ref 1 00000100:00100000:2.0:1789760793.077979:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484374912:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760793.078031:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25718784, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:2.0:1789760793.078041:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.078057:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:2.0:1789760793.078062:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760793.078261:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25718784->25722879] (req 25718784->25722879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.078276:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25718784->25722879] (req 25718784->25722879) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760793.111360:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973597 is committed 00002000:00010000:2.0:1789760793.111547:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25718784 -> 25722880 00010000:00010000:2.0:1789760793.111563:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25718784->25722879] (req 25718784->25722879) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.111576:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25718784->25722879] (req 25718784->25722879) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.111591:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25718784->25722879] (req 25718784->25722879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.111599:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760793.111605:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25718784->25722879] (req 25718784->25722879) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.111615:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25718784->25722879] (req 25718784->25722879) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.111633:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25718784->25722879] (req 25718784->25722879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.111641:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a56e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25718784->25722879] (req 25718784->25722879) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.111804:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484374912:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33823us (34048us total) trans 4294973597 rc 0/0 00000100:00100000:2.0:1789760793.111815:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6308 00000100:00100000:2.0:1789760793.111824:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782368824901 time 782408895884 state 0x2 ref 0 00000100:00100000:2.0:1789760793.124166:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760793.124500:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484375040 00000100:00100000:3.0:1789760793.124511:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484375040 00000100:00100000:3.0:1789760793.124566:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782368824901 time 782408895884 state 0x0 ref 1 00000100:00100000:3.0:1789760793.124581:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782409156300 time 782408895884 state 0x1 ref 1 00000100:00100000:3.0:1789760793.124610:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782409156300 time 782455540630 state 0x0 ref 1 00000100:00100000:3.0:1789760793.124622:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484375040:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760793.124670:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25722880, mtime: 0x6aad9518, atime: 0x0, ctime: 0x6aad9518, blocks: 0xc807 00010000:00010000:3.0:1789760793.124680:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.124694:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:3.0:1789760793.124698:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760793.124704:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25722880->25726975] (req 25722880->25726975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.124948:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25722880->25726975] (req 25722880->25726975) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760793.178952:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973598 is committed 00002000:00010000:0.0:1789760793.179256:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25722880 -> 25726976 00002000:00010000:0.0:1789760793.179261:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760792 -> 1789760793 00002000:00010000:0.0:1789760793.179264:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760792 -> 1789760793 00010000:00010000:0.0:1789760793.179275:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25722880->25726975] (req 25722880->25726975) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.179289:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25722880->25726975] (req 25722880->25726975) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.179299:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25722880->25726975] (req 25722880->25726975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.179306:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760793.179312:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25722880->25726975] (req 25722880->25726975) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.179320:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25722880->25726975] (req 25722880->25726975) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.179338:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25722880->25726975] (req 25722880->25726975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.179345:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a575 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25722880->25726975] (req 25722880->25726975) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760793.180313:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c45180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484375040:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54824us (55289us total) trans 4294973598 rc 0/0 00000100:00100000:0.0:1789760793.180325:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6309 00000100:00100000:0.0:1789760793.180336:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782409156300 time 782455540630 state 0x2 ref 0 00000100:00100000:0.0:1789760793.183634:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.184061:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484375168 00000100:00100000:1.0:1789760793.184106:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484375168 00000100:00100000:1.0:1789760793.184160:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782409156300 time 782455540630 state 0x0 ref 1 00000100:00100000:1.0:1789760793.184175:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782455801046 time 782455540630 state 0x1 ref 1 00000100:00100000:1.0:1789760793.184203:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782455801046 time 782515134512 state 0x0 ref 1 00000100:00100000:1.0:1789760793.184217:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484375168:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760793.184264:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25726976, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:1.0:1789760793.184273:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.184289:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:1.0:1789760793.184292:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760793.184298:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25726976->25731071] (req 25726976->25731071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.184311:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25726976->25731071] (req 25726976->25731071) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760793.215119:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25726976 -> 25731072 00010000:00010000:3.0:1789760793.215137:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25726976->25731071] (req 25726976->25731071) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.215151:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25726976->25731071] (req 25726976->25731071) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.215161:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25726976->25731071] (req 25726976->25731071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.215169:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760793.215173:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25726976->25731071] (req 25726976->25731071) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.215183:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25726976->25731071] (req 25726976->25731071) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.215204:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25726976->25731071] (req 25726976->25731071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.215212:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a57c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25726976->25731071] (req 25726976->25731071) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760793.215306:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9470e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484375168:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31086us (31676us total) trans 4294973599 rc 0/0 00000100:00100000:3.0:1789760793.215318:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6310 00000100:00100000:3.0:1789760793.215328:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782455801046 time 782515134512 state 0x2 ref 0 00000001:00080000:1.0:1789760793.215416:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973599 is committed 00000100:00100000:1.0:1789760793.219513:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760793.219659:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484375296 00000100:00100000:0.0:1789760793.219668:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484375296 00000100:00100000:0.0:1789760793.219733:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782455801046 time 782515134512 state 0x0 ref 1 00000100:00100000:0.0:1789760793.219748:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782515394928 time 782515134512 state 0x1 ref 1 00000100:00100000:0.0:1789760793.219777:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782515394928 time 782550708651 state 0x0 ref 1 00000100:00100000:0.0:1789760793.219789:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484375296:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760793.219835:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25731072, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:0.0:1789760793.219843:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.219857:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:0.0:1789760793.219861:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760793.219868:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25731072->25735167] (req 25731072->25735167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.219881:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25731072->25735167] (req 25731072->25735167) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760793.246778:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25731072 -> 25735168 00010000:00010000:0.0:1789760793.246798:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25731072->25735167] (req 25731072->25735167) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.246812:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25731072->25735167] (req 25731072->25735167) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.246821:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25731072->25735167] (req 25731072->25735167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.246828:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760793.246832:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25731072->25735167] (req 25731072->25735167) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.246842:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25731072->25735167] (req 25731072->25735167) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.246861:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25731072->25735167] (req 25731072->25735167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.246868:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a583 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25731072->25735167] (req 25731072->25735167) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760793.246977:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f12300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484375296:12345-192.168.203.10@tcp:4:dd.0 Request processed in 27184us (27468us total) trans 4294973600 rc 0/0 00000100:00100000:0.0:1789760793.246988:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6311 00000100:00100000:0.0:1789760793.246997:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782515394928 time 782550708651 state 0x2 ref 0 00000001:00080000:3.0:1789760793.247769:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973600 is committed 00000100:00100000:2.0:1789760793.251120:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760793.251227:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484375424 00000100:00100000:0.0:1789760793.251236:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484375424 00000100:00100000:0.0:1789760793.251294:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782515394928 time 782550708651 state 0x0 ref 1 00000100:00100000:0.0:1789760793.251308:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782550969067 time 782550708651 state 0x1 ref 1 00000100:00100000:0.0:1789760793.251337:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782550969067 time 782582268585 state 0x0 ref 1 00000100:00100000:0.0:1789760793.251350:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484375424:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760793.251399:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25735168, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:0.0:1789760793.251408:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.251424:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:0.0:1789760793.251430:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760793.251436:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25735168->25739263] (req 25735168->25739263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.251459:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25735168->25739263] (req 25735168->25739263) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760793.279677:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973601 is committed 00002000:00010000:0.0:1789760793.279850:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25735168 -> 25739264 00010000:00010000:0.0:1789760793.279866:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25735168->25739263] (req 25735168->25739263) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.279880:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25735168->25739263] (req 25735168->25739263) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.279889:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25735168->25739263] (req 25735168->25739263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.279897:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760793.279903:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25735168->25739263] (req 25735168->25739263) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.279913:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25735168->25739263] (req 25735168->25739263) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.279943:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25735168->25739263] (req 25735168->25739263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.279951:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a58a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25735168->25739263] (req 25735168->25739263) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760793.280135:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c7c47480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484375424:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28783us (29020us total) trans 4294973601 rc 0/0 00000100:00100000:0.0:1789760793.280146:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6312 00000100:00100000:0.0:1789760793.280155:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782550969067 time 782582268585 state 0x2 ref 0 00000100:00100000:2.0:1789760793.284601:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760793.284672:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484375552 00000100:00100000:3.0:1789760793.284681:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484375552 00000100:00100000:3.0:1789760793.284736:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782550969067 time 782582268585 state 0x0 ref 1 00000100:00100000:3.0:1789760793.284751:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782582529001 time 782582268585 state 0x1 ref 1 00000100:00100000:3.0:1789760793.284777:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782582529001 time 782615708258 state 0x0 ref 1 00000100:00100000:3.0:1789760793.284789:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484375552:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760793.284835:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25739264, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:3.0:1789760793.284844:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.284861:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:3.0:1789760793.284865:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760793.284871:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25739264->25743359] (req 25739264->25743359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.284885:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25739264->25743359] (req 25739264->25743359) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760793.309185:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973602 is committed 00002000:00010000:1.0:1789760793.309301:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25739264 -> 25743360 00010000:00010000:1.0:1789760793.309315:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25739264->25743359] (req 25739264->25743359) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.309329:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25739264->25743359] (req 25739264->25743359) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.309339:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25739264->25743359] (req 25739264->25743359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.309346:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760793.309352:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25739264->25743359] (req 25739264->25743359) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.309361:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25739264->25743359] (req 25739264->25743359) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.309381:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25739264->25743359] (req 25739264->25743359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.309388:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a591 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25739264->25743359] (req 25739264->25743359) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760793.309625:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484375552:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24832us (25029us total) trans 4294973602 rc 0/0 00000100:00100000:1.0:1789760793.309636:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6313 00000100:00100000:1.0:1789760793.309645:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782582529001 time 782615708258 state 0x2 ref 0 00000100:00100000:1.0:1789760793.313004:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760793.313110:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484375680 00000100:00100000:3.0:1789760793.313120:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484375680 00000100:00100000:3.0:1789760793.313172:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782582529001 time 782615708258 state 0x0 ref 1 00000100:00100000:3.0:1789760793.313186:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782615968674 time 782615708258 state 0x1 ref 1 00000100:00100000:3.0:1789760793.313207:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782615968674 time 782644137721 state 0x0 ref 1 00000100:00100000:3.0:1789760793.313219:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484375680:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760793.313263:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25743360, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:3.0:1789760793.313271:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.313287:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:3.0:1789760793.313292:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760793.313297:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25743360->25747455] (req 25743360->25747455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.313311:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25743360->25747455] (req 25743360->25747455) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760793.342744:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973603 is committed 00002000:00010000:3.0:1789760793.342874:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25743360 -> 25747456 00010000:00010000:3.0:1789760793.342889:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25743360->25747455] (req 25743360->25747455) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.342902:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25743360->25747455] (req 25743360->25747455) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.342911:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25743360->25747455] (req 25743360->25747455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.342919:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760793.342924:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25743360->25747455] (req 25743360->25747455) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.342942:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25743360->25747455] (req 25743360->25747455) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.342960:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25743360->25747455] (req 25743360->25747455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.342967:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a598 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25743360->25747455] (req 25743360->25747455) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760793.343049:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4f10700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484375680:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29828us (30050us total) trans 4294973603 rc 0/0 00000100:00100000:3.0:1789760793.343059:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6314 00000100:00100000:3.0:1789760793.343102:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782615968674 time 782644137721 state 0x2 ref 0 00000100:00100000:3.0:1789760793.346439:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760793.346958:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484375808 00000100:00100000:0.0:1789760793.346972:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484375808 00000100:00100000:0.0:1789760793.347032:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782615968674 time 782644137721 state 0x0 ref 1 00000100:00100000:0.0:1789760793.347048:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782644398137 time 782644137721 state 0x1 ref 1 00000100:00100000:0.0:1789760793.347228:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782644398137 time 782678157458 state 0x0 ref 1 00000100:00100000:0.0:1789760793.347244:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484375808:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760793.347290:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25747456, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:0.0:1789760793.347299:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.347317:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:0.0:1789760793.347321:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760793.347327:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25747456->25751551] (req 25747456->25751551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.347340:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25747456->25751551] (req 25747456->25751551) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760793.376390:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973604 is committed 00002000:00010000:3.0:1789760793.376464:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25747456 -> 25751552 00010000:00010000:3.0:1789760793.376478:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25747456->25751551] (req 25747456->25751551) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.376491:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25747456->25751551] (req 25747456->25751551) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.376499:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25747456->25751551] (req 25747456->25751551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.376571:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760793.376576:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25747456->25751551] (req 25747456->25751551) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.376585:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25747456->25751551] (req 25747456->25751551) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.376603:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25747456->25751551] (req 25747456->25751551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.376611:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a59f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25747456->25751551] (req 25747456->25751551) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760793.376728:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484375808:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29485us (30292us total) trans 4294973604 rc 0/0 00000100:00100000:3.0:1789760793.376737:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6315 00000100:00100000:3.0:1789760793.376746:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782644398137 time 782678157458 state 0x2 ref 0 00000100:00100000:2.0:1789760793.380163:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760793.380244:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484375936 00000100:00100000:0.0:1789760793.380253:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484375936 00000100:00100000:0.0:1789760793.380314:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782644398137 time 782678157458 state 0x0 ref 1 00000100:00100000:0.0:1789760793.380328:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782678417874 time 782678157458 state 0x1 ref 1 00000100:00100000:0.0:1789760793.380358:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782678417874 time 782711288784 state 0x0 ref 1 00000100:00100000:0.0:1789760793.380371:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484375936:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760793.380419:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25751552, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:0.0:1789760793.380428:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.380444:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:0.0:1789760793.380449:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760793.380455:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25751552->25755647] (req 25751552->25755647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.380468:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25751552->25755647] (req 25751552->25755647) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760793.407875:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973605 is committed 00002000:00010000:2.0:1789760793.408167:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25751552 -> 25755648 00010000:00010000:2.0:1789760793.408182:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25751552->25755647] (req 25751552->25755647) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.408195:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25751552->25755647] (req 25751552->25755647) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.408203:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25751552->25755647] (req 25751552->25755647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.408211:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760793.408215:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25751552->25755647] (req 25751552->25755647) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.408224:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25751552->25755647] (req 25751552->25755647) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.408244:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25751552->25755647] (req 25751552->25755647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.408253:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5a6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25751552->25755647] (req 25751552->25755647) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.408381:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484375936:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28008us (28221us total) trans 4294973605 rc 0/0 00000100:00100000:2.0:1789760793.408395:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6316 00000100:00100000:2.0:1789760793.408405:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782678417874 time 782711288784 state 0x2 ref 0 00000100:00100000:3.0:1789760793.412806:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.412943:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484376064 00000100:00100000:1.0:1789760793.412952:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484376064 00000100:00100000:1.0:1789760793.413001:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782678417874 time 782711288784 state 0x0 ref 1 00000100:00100000:1.0:1789760793.413030:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782711549200 time 782711288784 state 0x1 ref 1 00000100:00100000:1.0:1789760793.413292:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782711549200 time 782743992864 state 0x0 ref 1 00000100:00100000:1.0:1789760793.413311:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484376064:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760793.413359:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25755648, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:1.0:1789760793.413368:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.413383:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:1.0:1789760793.413388:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760793.413394:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25755648->25759743] (req 25755648->25759743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.413407:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25755648->25759743] (req 25755648->25759743) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760793.452143:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973606 is committed 00002000:00010000:3.0:1789760793.452337:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25755648 -> 25759744 00010000:00010000:3.0:1789760793.452352:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25755648->25759743] (req 25755648->25759743) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.452365:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25755648->25759743] (req 25755648->25759743) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.452374:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25755648->25759743] (req 25755648->25759743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.452382:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760793.452387:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25755648->25759743] (req 25755648->25759743) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.452396:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25755648->25759743] (req 25755648->25759743) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.452417:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25755648->25759743] (req 25755648->25759743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.452426:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a5ad lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25755648->25759743] (req 25755648->25759743) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760793.452602:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484376064:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39290us (39801us total) trans 4294973606 rc 0/0 00000100:00100000:3.0:1789760793.452613:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6317 00000100:00100000:3.0:1789760793.452621:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782711549200 time 782743992864 state 0x2 ref 0 00000100:00100000:3.0:1789760793.454980:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.455109:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484376192 00000100:00100000:1.0:1789760793.455119:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484376192 00000100:00100000:1.0:1789760793.455167:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782711549200 time 782743992864 state 0x0 ref 1 00000100:00100000:1.0:1789760793.455181:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782744253280 time 782743992864 state 0x1 ref 1 00000100:00100000:1.0:1789760793.455215:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782744253280 time 782786144903 state 0x0 ref 1 00000100:00100000:1.0:1789760793.455227:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484376192:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760793.455274:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25759744, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:1.0:1789760793.455282:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.455297:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:1.0:1789760793.455301:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760793.455307:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25759744->25763839] (req 25759744->25763839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.455320:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25759744->25763839] (req 25759744->25763839) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.484226:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760793.492402:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973607 is committed 00002000:00010000:0.0:1789760793.492861:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25759744 -> 25763840 00010000:00010000:0.0:1789760793.492877:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25759744->25763839] (req 25759744->25763839) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.492906:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25759744->25763839] (req 25759744->25763839) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.492916:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25759744->25763839] (req 25759744->25763839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.492922:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760793.492935:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25759744->25763839] (req 25759744->25763839) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.492944:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25759744->25763839] (req 25759744->25763839) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.492964:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25759744->25763839] (req 25759744->25763839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.492971:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a5b4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25759744->25763839] (req 25759744->25763839) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760793.493059:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484376192:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37830us (38082us total) trans 4294973607 rc 0/0 00000100:00100000:0.0:1789760793.493111:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6318 00000100:00100000:0.0:1789760793.493121:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782744253280 time 782786144903 state 0x2 ref 0 00000100:00100000:2.0:1789760793.505460:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760793.507133:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484376320 00000100:00100000:3.0:1789760793.507146:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484376320 00000100:00100000:3.0:1789760793.507209:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782744253280 time 782786144903 state 0x0 ref 1 00000100:00100000:3.0:1789760793.507225:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782786405319 time 782786144903 state 0x1 ref 1 00000100:00100000:3.0:1789760793.507259:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782786405319 time 782838190697 state 0x0 ref 1 00000100:00100000:3.0:1789760793.507270:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484376320:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760793.507319:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25763840, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:3.0:1789760793.507328:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.507344:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:3.0:1789760793.507348:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760793.507355:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25763840->25767935] (req 25763840->25767935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.507369:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25763840->25767935] (req 25763840->25767935) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760793.533596:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25763840 -> 25767936 00010000:00010000:3.0:1789760793.533616:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25763840->25767935] (req 25763840->25767935) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.533630:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25763840->25767935] (req 25763840->25767935) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.533639:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25763840->25767935] (req 25763840->25767935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.533648:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760793.533653:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25763840->25767935] (req 25763840->25767935) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.533662:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25763840->25767935] (req 25763840->25767935) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.533682:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25763840->25767935] (req 25763840->25767935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.533690:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5bb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25763840->25767935] (req 25763840->25767935) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760793.533870:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484376320:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26597us (28415us total) trans 4294973608 rc 0/0 00000100:00100000:3.0:1789760793.533882:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6319 00000100:00100000:3.0:1789760793.533893:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782786405319 time 782838190697 state 0x2 ref 0 00000001:00080000:3.0:1789760793.533940:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973608 is committed 00000100:00100000:1.0:1789760793.538792:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760793.539321:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484376448 00000100:00100000:2.0:1789760793.539331:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484376448 00000100:00100000:2.0:1789760793.539388:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782786405319 time 782838190697 state 0x0 ref 1 00000100:00100000:2.0:1789760793.539403:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782838451113 time 782838190697 state 0x1 ref 1 00000100:00100000:2.0:1789760793.539471:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782838451113 time 782870400936 state 0x0 ref 1 00000100:00100000:2.0:1789760793.539493:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484376448:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760793.539541:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25767936, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:2.0:1789760793.539550:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.539565:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:2.0:1789760793.539570:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760793.539577:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25767936->25772031] (req 25767936->25772031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.539591:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25767936->25772031] (req 25767936->25772031) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760793.572665:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973609 is committed 00002000:00010000:2.0:1789760793.572771:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25767936 -> 25772032 00010000:00010000:2.0:1789760793.572786:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25767936->25772031] (req 25767936->25772031) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.572799:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25767936->25772031] (req 25767936->25772031) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.572809:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25767936->25772031] (req 25767936->25772031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.572817:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760793.572823:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25767936->25772031] (req 25767936->25772031) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.572834:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25767936->25772031] (req 25767936->25772031) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.572853:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25767936->25772031] (req 25767936->25772031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.572862:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011808bef/0x2cf382639620a5c2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25767936->25772031] (req 25767936->25772031) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.573032:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484376448:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33537us (34250us total) trans 4294973609 rc 0/0 00000100:00100000:2.0:1789760793.573318:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6320 00000100:00100000:2.0:1789760793.573330:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782838451113 time 782870400936 state 0x2 ref 0 00000100:00100000:1.0:1789760793.577374:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760793.580444:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484376576 00000100:00100000:3.0:1789760793.580459:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484376576 00000100:00100000:3.0:1789760793.580524:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782838451113 time 782870400936 state 0x0 ref 1 00000100:00100000:3.0:1789760793.580608:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782870661352 time 782870400936 state 0x1 ref 1 00000100:00100000:3.0:1789760793.580700:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782870661352 time 782911619013 state 0x0 ref 1 00000100:00100000:3.0:1789760793.580724:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484376576:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760793.580775:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25772032, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:3.0:1789760793.580784:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.580800:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:3.0:1789760793.580805:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760793.580811:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25772032->25776127] (req 25772032->25776127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.580826:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25772032->25776127] (req 25772032->25776127) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760793.618259:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25772032 -> 25776128 00010000:00010000:3.0:1789760793.618278:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25772032->25776127] (req 25772032->25776127) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.618292:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25772032->25776127] (req 25772032->25776127) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.618301:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25772032->25776127] (req 25772032->25776127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.618309:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760793.618314:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25772032->25776127] (req 25772032->25776127) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.618324:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25772032->25776127] (req 25772032->25776127) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.618344:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25772032->25776127] (req 25772032->25776127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.618352:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5c9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25772032->25776127] (req 25772032->25776127) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760793.618442:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484376576:12345-192.168.203.10@tcp:4:dd.0 Request processed in 37718us (41073us total) trans 4294973610 rc 0/0 00000100:00100000:3.0:1789760793.618452:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6321 00000100:00100000:3.0:1789760793.618461:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782870661352 time 782911619013 state 0x2 ref 0 00000001:00080000:2.0:1789760793.618582:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973610 is committed 00000100:00100000:0.0:1789760793.622483:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.622569:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484376704 00000100:00100000:1.0:1789760793.622579:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484376704 00000100:00100000:1.0:1789760793.622634:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782870661352 time 782911619013 state 0x0 ref 1 00000100:00100000:1.0:1789760793.622650:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782911879429 time 782911619013 state 0x1 ref 1 00000100:00100000:1.0:1789760793.622755:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782911879429 time 782953685823 state 0x0 ref 1 00000100:00100000:1.0:1789760793.622769:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484376704:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760793.622817:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25776128, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:1.0:1789760793.622826:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.622842:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:1.0:1789760793.622847:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760793.622853:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25776128->25780223] (req 25776128->25780223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.622866:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25776128->25780223] (req 25776128->25780223) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760793.650444:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973611 is committed 00002000:00010000:1.0:1789760793.650580:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25776128 -> 25780224 00010000:00010000:1.0:1789760793.650595:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25776128->25780223] (req 25776128->25780223) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.650609:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25776128->25780223] (req 25776128->25780223) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.650618:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25776128->25780223] (req 25776128->25780223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.650626:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760793.650631:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25776128->25780223] (req 25776128->25780223) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.650640:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25776128->25780223] (req 25776128->25780223) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.650660:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25776128->25780223] (req 25776128->25780223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.650668:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a5d0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25776128->25780223] (req 25776128->25780223) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760793.650837:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484376704:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28066us (28358us total) trans 4294973611 rc 0/0 00000100:00100000:1.0:1789760793.650848:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6322 00000100:00100000:1.0:1789760793.650858:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782911879429 time 782953685823 state 0x2 ref 0 00000100:00100000:1.0:1789760793.658665:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760793.658803:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484376832 00000100:00100000:0.0:1789760793.658813:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484376832 00000100:00100000:0.0:1789760793.658873:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782911879429 time 782953685823 state 0x0 ref 1 00000100:00100000:0.0:1789760793.658888:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782953946239 time 782953685823 state 0x1 ref 1 00000100:00100000:0.0:1789760793.658963:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782953946239 time 782989893895 state 0x0 ref 1 00000100:00100000:0.0:1789760793.658977:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a32680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484376832:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760793.659036:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25780224, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:0.0:1789760793.659045:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.659060:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=782s v=(0 0) 00010000:00010000:0.0:1789760793.659087:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760793.659094:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25780224->25784319] (req 25780224->25784319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.659109:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25780224->25784319] (req 25780224->25784319) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760793.693688:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973612 is committed 00002000:00010000:2.0:1789760793.693880:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25780224 -> 25784320 00010000:00010000:2.0:1789760793.693895:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25780224->25784319] (req 25780224->25784319) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.693908:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25780224->25784319] (req 25780224->25784319) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.693917:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25780224->25784319] (req 25780224->25784319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.693934:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760793.693939:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25780224->25784319] (req 25780224->25784319) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.693948:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25780224->25784319] (req 25780224->25784319) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.693969:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25780224->25784319] (req 25780224->25784319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.693977:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a5d7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25780224->25784319] (req 25780224->25784319) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.694178:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a32680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484376832:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35199us (35519us total) trans 4294973612 rc 0/0 00000100:00100000:2.0:1789760793.694190:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6323 00000100:00100000:2.0:1789760793.694200:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782953946239 time 782989893895 state 0x2 ref 0 00000100:00100000:2.0:1789760793.697663:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760793.697811:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484376960 00000100:00100000:0.0:1789760793.697821:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484376960 00000100:00100000:0.0:1789760793.697877:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782953946239 time 782989893895 state 0x0 ref 1 00000100:00100000:0.0:1789760793.697891:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782990154311 time 782989893895 state 0x1 ref 1 00000100:00100000:0.0:1789760793.697981:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782990154311 time 783028911405 state 0x0 ref 1 00000100:00100000:0.0:1789760793.697997:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484376960:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760793.698311:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25784320, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:2.0:1789760793.698321:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.698337:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:2.0:1789760793.698341:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760793.698348:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25784320->25788415] (req 25784320->25788415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.698363:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25784320->25788415] (req 25784320->25788415) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760793.730282:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25784320 -> 25788416 00010000:00010000:2.0:1789760793.730301:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25784320->25788415] (req 25784320->25788415) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.730315:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25784320->25788415] (req 25784320->25788415) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.730330:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25784320->25788415] (req 25784320->25788415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.730337:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760793.730342:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25784320->25788415] (req 25784320->25788415) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.730350:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25784320->25788415] (req 25784320->25788415) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.730367:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25784320->25788415] (req 25784320->25788415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.730374:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a5de lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25784320->25788415] (req 25784320->25788415) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.730460:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1ed80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484376960:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32463us (32801us total) trans 4294973613 rc 0/0 00000100:00100000:2.0:1789760793.730470:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6324 00000100:00100000:2.0:1789760793.730480:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782990154311 time 783028911405 state 0x2 ref 0 00000100:00100000:0.0:1789760793.733645:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760793.733699:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484377088 00000100:00100000:3.0:1789760793.733709:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484377088 00000100:00100000:3.0:1789760793.733758:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 782990154311 time 783028911405 state 0x0 ref 1 00000100:00100000:3.0:1789760793.733770:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783029171821 time 783028911405 state 0x1 ref 1 00000100:00100000:3.0:1789760793.733842:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783029171821 time 783064771801 state 0x0 ref 1 00000100:00100000:3.0:1789760793.733856:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484377088:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760793.733905:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25788416, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:3.0:1789760793.733914:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.733940:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:3.0:1789760793.733945:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760793.733952:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25788416->25792511] (req 25788416->25792511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.733966:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25788416->25792511] (req 25788416->25792511) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760793.737599:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973613 is committed 00000001:00080000:0.0:1789760793.762261:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973614 is committed 00002000:00010000:3.0:1789760793.762395:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25788416 -> 25792512 00010000:00010000:3.0:1789760793.762416:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25788416->25792511] (req 25788416->25792511) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.762428:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25788416->25792511] (req 25788416->25792511) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.762445:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25788416->25792511] (req 25788416->25792511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.762456:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760793.762462:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25788416->25792511] (req 25788416->25792511) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.762471:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25788416->25792511] (req 25788416->25792511) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.762500:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25788416->25792511] (req 25788416->25792511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.762509:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5e5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25788416->25792511] (req 25788416->25792511) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760793.762599:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484377088:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28742us (28960us total) trans 4294973614 rc 0/0 00000100:00100000:3.0:1789760793.762609:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6325 00000100:00100000:3.0:1789760793.762617:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783029171821 time 783064771801 state 0x2 ref 0 00000100:00100000:0.0:1789760793.765846:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.765956:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484377216 00000100:00100000:1.0:1789760793.765966:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484377216 00000100:00100000:1.0:1789760793.766166:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783029171821 time 783064771801 state 0x0 ref 1 00000100:00100000:1.0:1789760793.766184:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783065032217 time 783064771801 state 0x1 ref 1 00000100:00100000:2.0:1789760793.766394:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783065032217 time 783097322395 state 0x0 ref 1 00000100:00100000:2.0:1789760793.766410:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484377216:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760793.766460:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25792512, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:2.0:1789760793.766469:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.766485:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:2.0:1789760793.766490:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760793.766496:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25792512->25796607] (req 25792512->25796607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.766510:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25792512->25796607] (req 25792512->25796607) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760793.795846:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973615 is committed 00002000:00010000:3.0:1789760793.795890:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25792512 -> 25796608 00010000:00010000:3.0:1789760793.795906:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25792512->25796607] (req 25792512->25796607) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.795919:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25792512->25796607] (req 25792512->25796607) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.796124:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25792512->25796607] (req 25792512->25796607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.796134:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760793.796139:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25792512->25796607] (req 25792512->25796607) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.796149:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25792512->25796607] (req 25792512->25796607) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.796315:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25792512->25796607] (req 25792512->25796607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.796323:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000be6cabda/0x2cf382639620a5ec lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25792512->25796607] (req 25792512->25796607) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.796494:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484377216:12345-192.168.203.10@tcp:4:dd.0 Request processed in 30082us (30650us total) trans 4294973615 rc 0/0 00000100:00100000:2.0:1789760793.796506:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6326 00000100:00100000:2.0:1789760793.796516:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783065032217 time 783097322395 state 0x2 ref 0 00000100:00100000:2.0:1789760793.799950:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760793.800050:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484377344 00000100:00100000:0.0:1789760793.800060:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484377344 00000100:00100000:0.0:1789760793.800147:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783065032217 time 783097322395 state 0x0 ref 1 00000100:00100000:0.0:1789760793.800163:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783097582811 time 783097322395 state 0x1 ref 1 00000100:00100000:3.0:1789760793.800213:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783097582811 time 783131141045 state 0x0 ref 1 00000100:00100000:3.0:1789760793.800237:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484377344:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760793.800285:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25796608, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:3.0:1789760793.800294:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.800310:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:3.0:1789760793.800315:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760793.800320:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25796608->25800703] (req 25796608->25800703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.800333:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25796608->25800703] (req 25796608->25800703) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760793.828978:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973616 is committed 00002000:00010000:2.0:1789760793.829214:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25796608 -> 25800704 00010000:00010000:2.0:1789760793.829288:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25796608->25800703] (req 25796608->25800703) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.829301:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25796608->25800703] (req 25796608->25800703) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.829311:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25796608->25800703] (req 25796608->25800703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.829319:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760793.829324:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25796608->25800703] (req 25796608->25800703) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.829333:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25796608->25800703] (req 25796608->25800703) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.829352:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25796608->25800703] (req 25796608->25800703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.829360:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a5f3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25796608->25800703] (req 25796608->25800703) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.829448:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484377344:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29211us (29504us total) trans 4294973616 rc 0/0 00000100:00100000:2.0:1789760793.829458:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6327 00000100:00100000:2.0:1789760793.829467:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783097582811 time 783131141045 state 0x2 ref 0 00000100:00100000:1.0:1789760793.832485:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.832517:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.832551:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.832580:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.832602:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760793.832625:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484377600 00000100:00100000:1.0:1789760793.832635:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484377600 00000100:00100000:1.0:1789760793.832682:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484377856 00000100:00100000:1.0:1789760793.832685:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484377856 00000100:00100000:1.0:1789760793.832700:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484377728 00000100:00100000:1.0:1789760793.832703:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484377728 00000100:00100000:2.0:1789760793.832786:0:12243:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484377984 00000100:00100000:2.0:1789760793.832791:0:12243:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484377984 00000100:00100000:2.0:1789760793.832817:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1021 00000100:00100000:2.0:1789760793.832827:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484377600:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760793.832893:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:32f439dd-5b05-4ca1-a11b-dd35989c33cd+8:2424:x1876699484377600:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 68us (418us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760793.832901:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1021 00000100:00100000:2.0:1789760793.832918:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1022 00000100:00100000:2.0:1789760793.832935:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a32d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484377856:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760793.832961:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a32d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484377856:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 37us (443us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760793.832967:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1022 00000100:00100000:2.0:1789760793.832978:0:12243:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1023 00000100:00100000:2.0:1789760793.832983:0:12243:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484377984:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:2.0:1789760793.833003:0:12243:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a31880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:2ccafb2e-9dad-4f46-90af-1cb0a60c36df+5:2424:x1876699484377984:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 21us (401us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760793.833020:0:12243:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1023 00000100:00080000:2.0:1789760793.833036:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760793.833040:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:1.0:1789760793.833410:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 1024 00000100:00100000:1.0:1789760793.833418:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f4a30e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484377728:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:1.0:1789760793.833450:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f4a30e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484377728:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 33us (869us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760793.833456:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 1024 00000100:00100000:0.0:1789760793.834091:0:6467:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484377472 00000100:00100000:0.0:1789760793.834097:0:6467:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484377472 00000100:00100000:0.0:1789760793.834145:0:6467:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.10@tcp, seq: 499 00000100:00100000:0.0:1789760793.834153:0:6467:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c803b050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484377472:12345-192.168.203.10@tcp:400:kworker.0 00000100:00100000:0.0:1789760793.834187:0:6467:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c803b050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:2627c619-a8ae-4613-a619-f7e733291af3+13:2424:x1876699484377472:12345-192.168.203.10@tcp:400:kworker.0 Request processed in 36us (1636us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760793.834194:0:6467:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.10@tcp, seq: 499 00000100:00080000:2.0:1789760793.835323:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00100000:2.0:1789760793.837371:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760793.837485:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484378112 00000100:00100000:0.0:1789760793.837495:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484378112 00000100:00100000:0.0:1789760793.837552:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783097582811 time 783131141045 state 0x0 ref 1 00000100:00100000:0.0:1789760793.837567:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783131401461 time 783131141045 state 0x1 ref 1 00000100:00100000:0.0:1789760793.837626:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783131401461 time 783168556965 state 0x0 ref 1 00000100:00100000:0.0:1789760793.837640:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484378112:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760793.837685:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25800704, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:0.0:1789760793.837693:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.837717:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:0.0:1789760793.837722:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760793.837728:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25800704->25804799] (req 25800704->25804799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.837742:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25800704->25804799] (req 25800704->25804799) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760793.873568:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973617 is committed 00002000:00010000:2.0:1789760793.873663:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25800704 -> 25804800 00010000:00010000:2.0:1789760793.873678:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25800704->25804799] (req 25800704->25804799) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.873690:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25800704->25804799] (req 25800704->25804799) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.873699:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25800704->25804799] (req 25800704->25804799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.873774:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760793.873780:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25800704->25804799] (req 25800704->25804799) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.873792:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25800704->25804799] (req 25800704->25804799) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.873812:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25800704->25804799] (req 25800704->25804799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.873821:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a5fa lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25800704->25804799] (req 25800704->25804799) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.873914:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1fb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484378112:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36272us (36547us total) trans 4294973617 rc 0/0 00000100:00100000:2.0:1789760793.873935:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6328 00000100:00100000:2.0:1789760793.873944:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783131401461 time 783168556965 state 0x2 ref 0 00000100:00100000:1.0:1789760793.878545:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760793.880435:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484378240 00000100:00100000:2.0:1789760793.880448:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484378240 00000100:00100000:3.0:1789760793.882501:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783131401461 time 783168556965 state 0x0 ref 1 00000100:00100000:3.0:1789760793.882523:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783168817381 time 783168556965 state 0x1 ref 1 00000100:00100000:3.0:1789760793.882621:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783168817381 time 783213551248 state 0x0 ref 1 00000100:00100000:3.0:1789760793.882636:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484378240:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760793.882687:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25804800, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:3.0:1789760793.882696:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.882722:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:3.0:1789760793.882726:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760793.882733:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25804800->25808895] (req 25804800->25808895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.882749:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25804800->25808895] (req 25804800->25808895) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760793.914357:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973618 is committed 00002000:00010000:3.0:1789760793.914467:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25804800 -> 25808896 00010000:00010000:3.0:1789760793.914482:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25804800->25808895] (req 25804800->25808895) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.914495:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25804800->25808895] (req 25804800->25808895) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.914505:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25804800->25808895] (req 25804800->25808895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.914513:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760793.914519:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25804800->25808895] (req 25804800->25808895) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.914530:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25804800->25808895] (req 25804800->25808895) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.914551:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25804800->25808895] (req 25804800->25808895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760793.914559:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a601 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25804800->25808895] (req 25804800->25808895) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760793.914651:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484378240:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32015us (36111us total) trans 4294973618 rc 0/0 00000100:00100000:3.0:1789760793.914661:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6329 00000100:00100000:3.0:1789760793.914671:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783168817381 time 783213551248 state 0x2 ref 0 00000100:00100000:1.0:1789760793.918487:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760793.918576:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484378368 00000100:00100000:2.0:1789760793.918586:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484378368 00000100:00100000:2.0:1789760793.918638:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783168817381 time 783213551248 state 0x0 ref 1 00000100:00100000:2.0:1789760793.918654:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783213811664 time 783213551248 state 0x1 ref 1 00000100:00100000:2.0:1789760793.918696:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783213811664 time 783249626437 state 0x0 ref 1 00000100:00100000:2.0:1789760793.918709:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484378368:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760793.918758:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25808896, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:2.0:1789760793.918768:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.918784:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:2.0:1789760793.918789:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760793.918795:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25808896->25812991] (req 25808896->25812991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.918810:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25808896->25812991] (req 25808896->25812991) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760793.949832:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973619 is committed 00002000:00010000:2.0:1789760793.950022:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25808896 -> 25812992 00010000:00010000:2.0:1789760793.950038:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25808896->25812991] (req 25808896->25812991) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.950052:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25808896->25812991] (req 25808896->25812991) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.950062:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25808896->25812991] (req 25808896->25812991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.950102:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760793.950107:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25808896->25812991] (req 25808896->25812991) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.950118:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25808896->25812991] (req 25808896->25812991) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.950143:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25808896->25812991] (req 25808896->25812991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.950151:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a608 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25808896->25812991] (req 25808896->25812991) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760793.950231:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484378368:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31519us (31748us total) trans 4294973619 rc 0/0 00000100:00100000:2.0:1789760793.950241:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6330 00000100:00100000:2.0:1789760793.950250:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783213811664 time 783249626437 state 0x2 ref 0 00000100:00100000:2.0:1789760793.953832:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760793.953934:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484378496 00000100:00100000:3.0:1789760793.953944:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484378496 00000100:00100000:3.0:1789760793.953998:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783213811664 time 783249626437 state 0x0 ref 1 00000100:00100000:3.0:1789760793.954137:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783249886853 time 783249626437 state 0x1 ref 1 00000100:00100000:2.0:1789760793.954264:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783249886853 time 783285193679 state 0x0 ref 1 00000100:00100000:2.0:1789760793.954279:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484378496:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760793.954327:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25812992, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:2.0:1789760793.954337:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.954352:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:2.0:1789760793.954357:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760793.954363:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25812992->25817087] (req 25812992->25817087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760793.954377:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25812992->25817087] (req 25812992->25817087) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760793.980510:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25812992 -> 25817088 00010000:00010000:1.0:1789760793.980529:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25812992->25817087] (req 25812992->25817087) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.980543:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25812992->25817087] (req 25812992->25817087) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.980551:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25812992->25817087] (req 25812992->25817087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.980558:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760793.980563:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25812992->25817087] (req 25812992->25817087) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.980572:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25812992->25817087] (req 25812992->25817087) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.980592:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25812992->25817087] (req 25812992->25817087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760793.980599:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a60f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25812992->25817087] (req 25812992->25817087) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760793.980701:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484378496:12345-192.168.203.10@tcp:4:dd.0 Request processed in 26421us (26875us total) trans 4294973620 rc 0/0 00000100:00100000:1.0:1789760793.980720:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6331 00000100:00100000:1.0:1789760793.980729:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783249886853 time 783285193679 state 0x2 ref 0 00000001:00080000:1.0:1789760793.980766:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973620 is committed 00000100:00100000:1.0:1789760793.984156:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760793.984235:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484378624 00000100:00100000:0.0:1789760793.984245:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484378624 00000100:00100000:0.0:1789760793.984301:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783249886853 time 783285193679 state 0x0 ref 1 00000100:00100000:0.0:1789760793.984316:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783285454095 time 783285193679 state 0x1 ref 1 00000100:00100000:0.0:1789760793.984346:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783285454095 time 783315276973 state 0x0 ref 1 00000100:00100000:0.0:1789760793.984360:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484378624:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760793.984406:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25817088, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:0.0:1789760793.984416:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.984435:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:0.0:1789760793.984439:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760793.984445:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25817088->25821183] (req 25817088->25821183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760793.984459:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25817088->25821183] (req 25817088->25821183) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760794.008914:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973621 is committed 00002000:00010000:0.0:1789760794.009051:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25817088 -> 25821184 00010000:00010000:0.0:1789760794.009085:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25817088->25821183] (req 25817088->25821183) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.009096:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25817088->25821183] (req 25817088->25821183) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.009102:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25817088->25821183] (req 25817088->25821183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.009107:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760794.009110:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25817088->25821183] (req 25817088->25821183) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.009117:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25817088->25821183] (req 25817088->25821183) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.009132:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25817088->25821183] (req 25817088->25821183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.009137:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a616 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25817088->25821183] (req 25817088->25821183) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760794.009225:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484378624:12345-192.168.203.10@tcp:4:dd.0 Request processed in 24864us (25201us total) trans 4294973621 rc 0/0 00000100:00100000:0.0:1789760794.009235:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6332 00000100:00100000:0.0:1789760794.009246:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783285454095 time 783315276973 state 0x2 ref 0 00000100:00100000:2.0:1789760794.013702:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760794.014310:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484378752 00000100:00100000:1.0:1789760794.014321:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484378752 00000100:00100000:1.0:1789760794.014390:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783285454095 time 783315276973 state 0x0 ref 1 00000100:00100000:1.0:1789760794.014404:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783315537389 time 783315276973 state 0x1 ref 1 00000100:00100000:1.0:1789760794.014482:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783315537389 time 783345412143 state 0x0 ref 1 00000100:00100000:1.0:1789760794.014496:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484378752:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760794.014546:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25821184, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:1.0:1789760794.014554:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.014570:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:1.0:1789760794.014574:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760794.014581:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25821184->25825279] (req 25821184->25825279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.014594:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25821184->25825279] (req 25821184->25825279) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1789760794.015114:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:3.0:1789760794.069664:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973622 is committed 00002000:00010000:1.0:1789760794.069810:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25821184 -> 25825280 00010000:00010000:1.0:1789760794.069825:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25821184->25825279] (req 25821184->25825279) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.069837:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25821184->25825279] (req 25821184->25825279) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.069846:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25821184->25825279] (req 25821184->25825279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.069853:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.069858:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25821184->25825279] (req 25821184->25825279) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.069867:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25821184->25825279] (req 25821184->25825279) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.069888:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25821184->25825279] (req 25821184->25825279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.069895:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a61d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25821184->25825279] (req 25821184->25825279) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.070005:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484378752:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55507us (56306us total) trans 4294973622 rc 0/0 00000100:00100000:1.0:1789760794.070016:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6333 00000100:00100000:1.0:1789760794.070027:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783315537389 time 783345412143 state 0x2 ref 0 00000100:00100000:2.0:1789760794.077492:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760794.077666:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484378880 00000100:00100000:0.0:1789760794.077685:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484378880 00000100:00100000:0.0:1789760794.077757:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783315537389 time 783345412143 state 0x0 ref 1 00000100:00100000:0.0:1789760794.077772:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783345672559 time 783345412143 state 0x1 ref 1 00000100:00100000:0.0:1789760794.077802:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783345672559 time 783408733490 state 0x0 ref 1 00000100:00100000:0.0:1789760794.077816:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9653b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484378880:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760794.077870:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25825280, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:0.0:1789760794.077879:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.077895:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:0.0:1789760794.077899:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760794.077904:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25825280->25829375] (req 25825280->25829375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.077919:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25825280->25829375] (req 25825280->25829375) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760794.121238:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25825280 -> 25829376 00010000:00010000:3.0:1789760794.121257:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25825280->25829375] (req 25825280->25829375) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.121276:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25825280->25829375] (req 25825280->25829375) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.121284:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25825280->25829375] (req 25825280->25829375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.121291:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760794.121295:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25825280->25829375] (req 25825280->25829375) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.121303:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25825280->25829375] (req 25825280->25829375) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.121321:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25825280->25829375] (req 25825280->25829375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.121328:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a624 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25825280->25829375] (req 25825280->25829375) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760794.121453:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9653b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484378880:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43636us (43966us total) trans 4294973623 rc 0/0 00000100:00100000:3.0:1789760794.121463:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6334 00000100:00100000:3.0:1789760794.121472:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783345672559 time 783408733490 state 0x2 ref 0 00000001:00080000:1.0:1789760794.122210:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973623 is committed 00000100:00100000:2.0:1789760794.124767:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760794.124813:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484379008 00000100:00100000:3.0:1789760794.124823:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484379008 00000100:00100000:3.0:1789760794.124872:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783345672559 time 783408733490 state 0x0 ref 1 00000100:00100000:3.0:1789760794.124888:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783408993906 time 783408733490 state 0x1 ref 1 00000100:00100000:3.0:1789760794.124933:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783408993906 time 783455862623 state 0x0 ref 1 00000100:00100000:3.0:1789760794.124946:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484379008:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760794.125060:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25829376, mtime: 0x6aad9519, atime: 0x0, ctime: 0x6aad9519, blocks: 0xc807 00010000:00010000:3.0:1789760794.125144:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.125161:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:3.0:1789760794.125166:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760794.125172:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25829376->25833471] (req 25829376->25833471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.125185:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25829376->25833471] (req 25829376->25833471) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760794.170015:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25829376 -> 25833472 00002000:00010000:2.0:1789760794.170025:0:8106:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760793 -> 1789760794 00002000:00010000:2.0:1789760794.170027:0:8106:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760793 -> 1789760794 00010000:00010000:2.0:1789760794.170041:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25829376->25833471] (req 25829376->25833471) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.170056:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25829376->25833471] (req 25829376->25833471) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.170084:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25829376->25833471] (req 25829376->25833471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.170094:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760794.170099:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25829376->25833471] (req 25829376->25833471) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.170109:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25829376->25833471] (req 25829376->25833471) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760794.170121:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973624 is committed 00010000:00010000:2.0:1789760794.170129:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25829376->25833471] (req 25829376->25833471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.170153:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a62b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25829376->25833471] (req 25829376->25833471) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760794.170306:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9652d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484379008:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45357us (45542us total) trans 4294973624 rc 0/0 00000100:00100000:2.0:1789760794.170316:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6335 00000100:00100000:2.0:1789760794.170326:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783408993906 time 783455862623 state 0x2 ref 0 00000100:00100000:2.0:1789760794.174523:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760794.174981:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484379136 00000100:00100000:0.0:1789760794.175001:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484379136 00000100:00100000:0.0:1789760794.175061:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783408993906 time 783455862623 state 0x0 ref 1 00000100:00100000:0.0:1789760794.175105:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783456123039 time 783455862623 state 0x1 ref 1 00000100:00100000:0.0:1789760794.175182:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783456123039 time 783506111370 state 0x0 ref 1 00000100:00100000:0.0:1789760794.175197:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484379136:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760794.175344:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25833472, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760794.175355:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.175370:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:1.0:1789760794.175375:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760794.175381:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25833472->25837567] (req 25833472->25837567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.175395:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25833472->25837567] (req 25833472->25837567) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760794.234715:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973625 is committed 00002000:00010000:2.0:1789760794.237893:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25833472 -> 25837568 00010000:00010000:2.0:1789760794.237914:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25833472->25837567] (req 25833472->25837567) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.237940:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25833472->25837567] (req 25833472->25837567) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.237950:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25833472->25837567] (req 25833472->25837567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.237959:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760794.237964:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25833472->25837567] (req 25833472->25837567) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.237973:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25833472->25837567] (req 25833472->25837567) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.238187:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25833472->25837567] (req 25833472->25837567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.238196:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b7926345/0x2cf382639620a632 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25833472->25837567] (req 25833472->25837567) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760794.238300:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9650a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484379136:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63102us (63781us total) trans 4294973625 rc 0/0 00000100:00100000:2.0:1789760794.238311:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6336 00000100:00100000:2.0:1789760794.238321:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783456123039 time 783506111370 state 0x2 ref 0 00000100:00100000:2.0:1789760794.256423:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760794.256501:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484379264 00000100:00100000:1.0:1789760794.256511:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484379264 00000100:00100000:1.0:1789760794.256564:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783456123039 time 783506111370 state 0x0 ref 1 00000100:00100000:1.0:1789760794.256579:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783506371786 time 783506111370 state 0x1 ref 1 00000100:00100000:1.0:1789760794.256608:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783506371786 time 783587539107 state 0x0 ref 1 00000100:00100000:1.0:1789760794.256621:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9651f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484379264:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760794.256668:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25837568, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760794.256677:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.256692:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:1.0:1789760794.256697:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760794.256704:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25837568->25841663] (req 25837568->25841663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.256718:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25837568->25841663] (req 25837568->25841663) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760794.303497:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25837568 -> 25841664 00010000:00010000:1.0:1789760794.303516:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25837568->25841663] (req 25837568->25841663) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.303531:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25837568->25841663] (req 25837568->25841663) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.303547:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25837568->25841663] (req 25837568->25841663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.303555:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.303561:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25837568->25841663] (req 25837568->25841663) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.303570:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25837568->25841663] (req 25837568->25841663) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.303589:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25837568->25841663] (req 25837568->25841663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.303597:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a639 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25837568->25841663] (req 25837568->25841663) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.303691:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9651f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484379264:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47068us (47272us total) trans 4294973626 rc 0/0 00000100:00100000:1.0:1789760794.305220:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6337 00000100:00100000:1.0:1789760794.305374:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783506371786 time 783587539107 state 0x2 ref 0 00000001:00080000:3.0:1789760794.307363:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973626 is committed 00000100:00100000:1.0:1789760794.310122:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760794.310270:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484379392 00000100:00100000:3.0:1789760794.310280:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484379392 00000100:00100000:3.0:1789760794.310334:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783506371786 time 783587539107 state 0x0 ref 1 00000100:00100000:3.0:1789760794.310350:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783587799523 time 783587539107 state 0x1 ref 1 00000100:00100000:3.0:1789760794.310372:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783587799523 time 783641303174 state 0x0 ref 1 00000100:00100000:3.0:1789760794.310384:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484379392:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760794.310432:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25841664, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:3.0:1789760794.310441:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.310457:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:3.0:1789760794.310461:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760794.310467:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25841664->25845759] (req 25841664->25845759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.310493:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25841664->25845759] (req 25841664->25845759) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760794.350026:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973627 is committed 00002000:00010000:1.0:1789760794.350126:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25841664 -> 25845760 00010000:00010000:1.0:1789760794.350140:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25841664->25845759] (req 25841664->25845759) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.350160:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25841664->25845759] (req 25841664->25845759) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.350170:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25841664->25845759] (req 25841664->25845759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.350177:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.350182:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25841664->25845759] (req 25841664->25845759) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.350191:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25841664->25845759] (req 25841664->25845759) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.350211:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25841664->25845759] (req 25841664->25845759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.350219:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a640 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25841664->25845759] (req 25841664->25845759) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.350315:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484379392:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39928us (40198us total) trans 4294973627 rc 0/0 00000100:00100000:1.0:1789760794.350325:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6338 00000100:00100000:1.0:1789760794.350335:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783587799523 time 783641303174 state 0x2 ref 0 00000100:00100000:2.0:1789760794.356212:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760794.356773:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484379520 00000100:00100000:3.0:1789760794.356784:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484379520 00000100:00100000:2.0:1789760794.359344:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783587799523 time 783641303174 state 0x0 ref 1 00000100:00100000:2.0:1789760794.359364:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783641563590 time 783641303174 state 0x1 ref 1 00000100:00100000:2.0:1789760794.359405:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783641563590 time 783690335525 state 0x0 ref 1 00000100:00100000:2.0:1789760794.359427:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484379520:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760794.359800:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25845760, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:2.0:1789760794.359811:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.359842:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:2.0:1789760794.359847:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760794.359853:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25845760->25849855] (req 25845760->25849855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.359867:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25845760->25849855] (req 25845760->25849855) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760794.403649:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25845760 -> 25849856 00010000:00010000:2.0:1789760794.403668:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25845760->25849855] (req 25845760->25849855) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.403681:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25845760->25849855] (req 25845760->25849855) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.403689:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25845760->25849855] (req 25845760->25849855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.403696:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760794.403701:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25845760->25849855] (req 25845760->25849855) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.403721:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25845760->25849855] (req 25845760->25849855) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.403741:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25845760->25849855] (req 25845760->25849855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.403749:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a647 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25845760->25849855] (req 25845760->25849855) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760794.403843:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfbb6300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484379520:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44422us (47634us total) trans 4294973628 rc 0/0 00000100:00100000:2.0:1789760794.403855:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6339 00000100:00100000:2.0:1789760794.403865:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783641563590 time 783690335525 state 0x2 ref 0 00000001:00080000:2.0:1789760794.403913:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973628 is committed 00000100:00100000:2.0:1789760794.408906:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760794.409058:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484379648 00000100:00100000:1.0:1789760794.409109:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484379648 00000100:00100000:1.0:1789760794.409164:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783641563590 time 783690335525 state 0x0 ref 1 00000100:00100000:1.0:1789760794.409181:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783690595941 time 783690335525 state 0x1 ref 1 00000100:00100000:1.0:1789760794.409212:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783690595941 time 783740142695 state 0x0 ref 1 00000100:00100000:1.0:1789760794.409432:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484379648:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760794.409805:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25849856, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760794.409816:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.409832:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:1.0:1789760794.409837:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760794.409842:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25849856->25853951] (req 25849856->25853951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.409856:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25849856->25853951] (req 25849856->25853951) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760794.459756:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973629 is committed 00002000:00010000:1.0:1789760794.459935:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25849856 -> 25853952 00010000:00010000:1.0:1789760794.459951:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25849856->25853951] (req 25849856->25853951) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.459963:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25849856->25853951] (req 25849856->25853951) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.459977:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25849856->25853951] (req 25849856->25853951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.459985:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.459989:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25849856->25853951] (req 25849856->25853951) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.459998:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25849856->25853951] (req 25849856->25853951) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.460100:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25849856->25853951] (req 25849856->25853951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.460109:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a64e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25849856->25853951] (req 25849856->25853951) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.460206:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484379648:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50978us (51305us total) trans 4294973629 rc 0/0 00000100:00100000:1.0:1789760794.460217:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6340 00000100:00100000:1.0:1789760794.460296:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783690595941 time 783740142695 state 0x2 ref 0 00000100:00100000:0.0:1789760794.463167:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760794.463399:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484379776 00000100:00100000:1.0:1789760794.463409:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484379776 00000100:00100000:1.0:1789760794.463697:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783690595941 time 783740142695 state 0x0 ref 1 00000100:00100000:1.0:1789760794.463725:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783740403111 time 783740142695 state 0x1 ref 1 00000100:00100000:1.0:1789760794.463761:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783740403111 time 783794691900 state 0x0 ref 1 00000100:00100000:1.0:1789760794.463774:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484379776:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760794.463824:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25853952, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760794.463833:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.463849:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:1.0:1789760794.463854:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760794.463860:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25853952->25858047] (req 25853952->25858047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.463874:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25853952->25858047] (req 25853952->25858047) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760794.508260:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8109500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508945536:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1789760794.508458:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760794.508611:0:11334:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508945536 00000100:00100000:0.0:1789760794.508618:0:11334:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508945536 00000100:00100000:0.0:1789760794.508681:0:11334:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 422 00000100:00100000:0.0:1789760794.508691:0:11334:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abb100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508945536:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:0.0:1789760794.508724:0:11334:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528023552 pending 1703936 free 3832545280 avail 3812622336 00000100:00100000:0.0:1789760794.508766:0:11334:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abb100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3481:x1876699508945536:12345-0@lo:13:osp-pre-0-0.0 Request processed in 77us (308us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760794.508774:0:11334:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 422 00000100:00080000:3.0:1789760794.508882:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760794.508930:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8109500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3481:1876699508945536:0@lo:13:osp-pre-0-0.0 00010000:00010000:1.0:1789760794.509054:0:8118:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760794.509060:0:3479:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760794.509086:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c15c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508945664:0@lo:41:osp-pre-0-1.0 00000100:00100000:1.0:1789760794.509119:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760794.509153:0:6478:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508945664 00000100:00100000:1.0:1789760794.509157:0:6478:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508945664 00000100:00100000:1.0:1789760794.509186:0:6478:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 481 00000100:00100000:1.0:1789760794.509192:0:6478:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508945664:12345-0@lo:41:osp-pre-0-1.0 00000004:00000024:1.0:1789760794.509387:0:6478:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261647360 avail 2259550208 00000100:00100000:1.0:1789760794.509438:0:6478:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15880 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:lustre-MDT0001-mdtlov_UUID+6:3479:x1876699508945664:12345-0@lo:41:osp-pre-0-1.0 Request processed in 245us (317us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760794.509447:0:6478:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 481 00000100:00100000:3.0:1789760794.509521:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c15c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508945664:0@lo:41:osp-pre-0-1.0 00000100:00100000:3.0:1789760794.509647:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c8109500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508945792:0@lo:13:osp-pre-0-1.0 00000100:00100000:3.0:1789760794.509676:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1789760794.509813:0:8105:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508945792 00000100:00100000:0.0:1789760794.509818:0:8105:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508945792 00000100:00100000:0.0:1789760794.509863:0:8105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 423 00000100:00100000:0.0:1789760794.509871:0:8105:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8aba680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508945792:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:0.0:1789760794.509884:0:8105:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 528023552 pending 1703936 free 3832545280 avail 3812622336 00000100:00100000:0.0:1789760794.509916:0:8105:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8aba680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3479:x1876699508945792:12345-0@lo:13:osp-pre-0-1.0 Request processed in 47us (240us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760794.509936:0:8105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 423 00000100:00080000:3.0:1789760794.510026:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760794.510049:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c8109500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0001-mdtlov_UUID:3479:1876699508945792:0@lo:13:osp-pre-0-1.0 00000001:00080000:0.0:1789760794.511019:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973630 is committed 00002000:00010000:1.0:1789760794.511114:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25853952 -> 25858048 00010000:00010000:1.0:1789760794.511128:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25853952->25858047] (req 25853952->25858047) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.511141:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25853952->25858047] (req 25853952->25858047) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.511150:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25853952->25858047] (req 25853952->25858047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.511158:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.511163:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25853952->25858047] (req 25853952->25858047) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.511173:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25853952->25858047] (req 25853952->25858047) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.511192:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25853952->25858047] (req 25853952->25858047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.511201:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a655 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25853952->25858047] (req 25853952->25858047) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.511362:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484379776:12345-192.168.203.10@tcp:4:dd.0 Request processed in 47587us (48200us total) trans 4294973630 rc 0/0 00000100:00100000:1.0:1789760794.511376:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6341 00000100:00100000:1.0:1789760794.511385:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783740403111 time 783794691900 state 0x2 ref 0 00000100:00100000:2.0:1789760794.514716:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760794.514806:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484379904 00000100:00100000:3.0:1789760794.514816:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484379904 00000100:00100000:3.0:1789760794.514868:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783740403111 time 783794691900 state 0x0 ref 1 00000100:00100000:3.0:1789760794.514882:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783794952316 time 783794691900 state 0x1 ref 1 00000100:00100000:3.0:1789760794.514911:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783794952316 time 783845842633 state 0x0 ref 1 00000100:00100000:3.0:1789760794.514934:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484379904:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760794.514980:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25858048, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:3.0:1789760794.514989:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.515004:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:3.0:1789760794.515009:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760794.515015:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25858048->25862143] (req 25858048->25862143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.515028:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25858048->25862143] (req 25858048->25862143) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760794.554385:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973631 is committed 00002000:00010000:1.0:1789760794.554499:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25858048 -> 25862144 00010000:00010000:1.0:1789760794.554514:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25858048->25862143] (req 25858048->25862143) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.554527:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25858048->25862143] (req 25858048->25862143) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.554537:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25858048->25862143] (req 25858048->25862143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.554544:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.554549:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25858048->25862143] (req 25858048->25862143) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.554557:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25858048->25862143] (req 25858048->25862143) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.554576:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25858048->25862143] (req 25858048->25862143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.554584:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a65c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25858048->25862143] (req 25858048->25862143) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.554779:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484379904:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39843us (40067us total) trans 4294973631 rc 0/0 00000100:00100000:1.0:1789760794.554791:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6342 00000100:00100000:1.0:1789760794.554801:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783794952316 time 783845842633 state 0x2 ref 0 00000100:00100000:2.0:1789760794.563161:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760794.563723:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484380032 00000100:00100000:3.0:1789760794.563734:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484380032 00000100:00100000:3.0:1789760794.563787:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783794952316 time 783845842633 state 0x0 ref 1 00000100:00100000:3.0:1789760794.563802:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783846103049 time 783845842633 state 0x1 ref 1 00000100:00100000:3.0:1789760794.563839:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783846103049 time 783894770315 state 0x0 ref 1 00000100:00100000:3.0:1789760794.563854:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484380032:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760794.563904:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25862144, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:3.0:1789760794.563913:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.564215:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:3.0:1789760794.564219:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760794.564226:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25862144->25866239] (req 25862144->25866239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.564241:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25862144->25866239] (req 25862144->25866239) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760794.613374:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973632 is committed 00002000:00010000:3.0:1789760794.613641:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25862144 -> 25866240 00010000:00010000:3.0:1789760794.613655:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25862144->25866239] (req 25862144->25866239) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.613668:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25862144->25866239] (req 25862144->25866239) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.613676:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25862144->25866239] (req 25862144->25866239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.613684:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760794.613689:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25862144->25866239] (req 25862144->25866239) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.613698:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25862144->25866239] (req 25862144->25866239) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.613717:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25862144->25866239] (req 25862144->25866239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.613726:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a663 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25862144->25866239] (req 25862144->25866239) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760794.613810:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484380032:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49956us (50655us total) trans 4294973632 rc 0/0 00000100:00100000:3.0:1789760794.613819:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6343 00000100:00100000:3.0:1789760794.613827:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783846103049 time 783894770315 state 0x2 ref 0 00000100:00100000:1.0:1789760794.622852:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760794.622971:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484380160 00000100:00100000:2.0:1789760794.622982:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484380160 00000100:00100000:2.0:1789760794.623034:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783846103049 time 783894770315 state 0x0 ref 1 00000100:00100000:2.0:1789760794.623049:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783895030731 time 783894770315 state 0x1 ref 1 00000100:00100000:2.0:1789760794.623120:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783895030731 time 783954050809 state 0x0 ref 1 00000100:00100000:2.0:1789760794.623133:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484380160:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760794.623185:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25866240, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:2.0:1789760794.623195:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.623217:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=783s v=(0 0) 00010000:00010000:2.0:1789760794.623221:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760794.623228:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25866240->25870335] (req 25866240->25870335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.623241:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25866240->25870335] (req 25866240->25870335) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760794.667342:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973633 is committed 00002000:00010000:1.0:1789760794.667449:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25866240 -> 25870336 00010000:00010000:1.0:1789760794.667466:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25866240->25870335] (req 25866240->25870335) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.667480:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25866240->25870335] (req 25866240->25870335) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.667490:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25866240->25870335] (req 25866240->25870335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.667498:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.667503:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25866240->25870335] (req 25866240->25870335) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.667513:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25866240->25870335] (req 25866240->25870335) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.667532:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25866240->25870335] (req 25866240->25870335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.667546:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f08156ae/0x2cf382639620a66a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25866240->25870335] (req 25866240->25870335) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.667642:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484380160:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44506us (44794us total) trans 4294973633 rc 0/0 00000100:00100000:1.0:1789760794.667652:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6344 00000100:00100000:1.0:1789760794.667662:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783895030731 time 783954050809 state 0x2 ref 0 00000100:00100000:2.0:1789760794.672638:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760794.676859:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484380288 00000100:00100000:1.0:1789760794.676873:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484380288 00000100:00100000:1.0:1789760794.676940:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783895030731 time 783954050809 state 0x0 ref 1 00000100:00100000:1.0:1789760794.676955:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783954311225 time 783954050809 state 0x1 ref 1 00000100:00100000:1.0:1789760794.677030:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783954311225 time 784007960172 state 0x0 ref 1 00000100:00100000:1.0:1789760794.677044:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484380288:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760794.677113:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25870336, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760794.677122:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.677138:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760794.677142:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760794.677148:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25870336->25874431] (req 25870336->25874431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.677168:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25870336->25874431] (req 25870336->25874431) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760794.719369:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25870336 -> 25874432 00010000:00010000:1.0:1789760794.719389:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25870336->25874431] (req 25870336->25874431) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.719401:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25870336->25874431] (req 25870336->25874431) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.719411:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25870336->25874431] (req 25870336->25874431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.719419:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfc00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.719424:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25870336->25874431] (req 25870336->25874431) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.719434:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25870336->25874431] (req 25870336->25874431) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.719454:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25870336->25874431] (req 25870336->25874431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.719463:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000fc72f95e/0x2cf382639620a671 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25870336->25874431] (req 25870336->25874431) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.719624:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501ba0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484380288:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42579us (46991us total) trans 4294973634 rc 0/0 00000100:00100000:1.0:1789760794.719636:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6345 00000100:00100000:1.0:1789760794.719645:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783954311225 time 784007960172 state 0x2 ref 0 00000001:00080000:3.0:1789760794.720938:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973634 is committed 00000100:00100000:3.0:1789760794.723655:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760794.723742:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484380416 00000100:00100000:1.0:1789760794.723820:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484380416 00000100:00100000:1.0:1789760794.723874:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 783954311225 time 784007960172 state 0x0 ref 1 00000100:00100000:1.0:1789760794.723888:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784008220588 time 784007960172 state 0x1 ref 1 00000100:00100000:1.0:1789760794.723934:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784008220588 time 784054863460 state 0x0 ref 1 00000100:00100000:1.0:1789760794.723946:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484380416:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760794.724031:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25874432, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760794.724040:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.724057:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760794.724232:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760794.724241:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25874432->25878527] (req 25874432->25878527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.724256:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25874432->25878527] (req 25874432->25878527) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760794.758891:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973635 is committed 00002000:00010000:3.0:1789760794.758971:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25874432 -> 25878528 00010000:00010000:3.0:1789760794.758987:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25874432->25878527] (req 25874432->25878527) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.759005:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25874432->25878527] (req 25874432->25878527) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.759015:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25874432->25878527] (req 25874432->25878527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.759024:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfe00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760794.759030:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25874432->25878527] (req 25874432->25878527) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.759043:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25874432->25878527] (req 25874432->25878527) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.759094:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25874432->25878527] (req 25874432->25878527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.759102:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002d7349f5/0x2cf382639620a678 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25874432->25878527] (req 25874432->25878527) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760794.760016:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484380416:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36064us (36361us total) trans 4294973635 rc 0/0 00000100:00100000:2.0:1789760794.760028:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6346 00000100:00100000:2.0:1789760794.760038:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784008220588 time 784054863460 state 0x2 ref 0 00000100:00100000:2.0:1789760794.767211:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760794.767602:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484380544 00000100:00100000:3.0:1789760794.767612:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484380544 00000100:00100000:3.0:1789760794.767667:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784008220588 time 784054863460 state 0x0 ref 1 00000100:00100000:3.0:1789760794.767682:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784055123876 time 784054863460 state 0x1 ref 1 00000100:00100000:1.0:1789760794.767750:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784055123876 time 784098679058 state 0x0 ref 1 00000100:00100000:1.0:1789760794.767765:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484380544:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760794.767812:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25878528, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760794.767821:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.767837:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760794.767842:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760794.767849:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25878528->25882623] (req 25878528->25882623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.767863:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25878528->25882623] (req 25878528->25882623) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760794.821767:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25878528 -> 25882624 00000001:00080000:1.0:1789760794.821767:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973636 is committed 00010000:00010000:0.0:1789760794.821785:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25878528->25882623] (req 25878528->25882623) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.821800:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25878528->25882623] (req 25878528->25882623) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.821809:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25878528->25882623] (req 25878528->25882623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.821816:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760794.821820:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25878528->25882623] (req 25878528->25882623) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.821829:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25878528->25882623] (req 25878528->25882623) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.821850:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25878528->25882623] (req 25878528->25882623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.821862:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a67f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25878528->25882623] (req 25878528->25882623) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760794.822178:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484380544:12345-192.168.203.10@tcp:4:dd.0 Request processed in 54411us (54972us total) trans 4294973636 rc 0/0 00000100:00100000:0.0:1789760794.822189:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6347 00000100:00100000:0.0:1789760794.822198:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784055123876 time 784098679058 state 0x2 ref 0 00000100:00100000:1.0:1789760794.827765:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760794.827990:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484380672 00000100:00100000:2.0:1789760794.828000:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484380672 00000100:00100000:2.0:1789760794.828054:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784055123876 time 784098679058 state 0x0 ref 1 00000100:00100000:2.0:1789760794.828142:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784098939474 time 784098679058 state 0x1 ref 1 00000100:00100000:2.0:1789760794.828220:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784098939474 time 784159150690 state 0x0 ref 1 00000100:00100000:2.0:1789760794.828234:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484380672:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760794.828282:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25882624, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:2.0:1789760794.828292:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.828308:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:2.0:1789760794.828313:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760794.828319:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25882624->25886719] (req 25882624->25886719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760794.828332:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25882624->25886719] (req 25882624->25886719) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760794.873023:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973637 is committed 00002000:00010000:1.0:1789760794.873131:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25882624 -> 25886720 00010000:00010000:1.0:1789760794.873146:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25882624->25886719] (req 25882624->25886719) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.873159:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25882624->25886719] (req 25882624->25886719) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.873168:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25882624->25886719] (req 25882624->25886719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.873176:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.873182:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25882624->25886719] (req 25882624->25886719) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.873191:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25882624->25886719] (req 25882624->25886719) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.873213:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25882624->25886719] (req 25882624->25886719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.873223:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a686 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25882624->25886719] (req 25882624->25886719) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.873317:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484380672:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45081us (45557us total) trans 4294973637 rc 0/0 00000100:00100000:1.0:1789760794.873328:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6348 00000100:00100000:1.0:1789760794.873338:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784098939474 time 784159150690 state 0x2 ref 0 00000100:00100000:1.0:1789760794.876947:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760794.877108:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484380800 00000100:00100000:3.0:1789760794.877128:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484380800 00000100:00100000:3.0:1789760794.877179:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784098939474 time 784159150690 state 0x0 ref 1 00000100:00100000:3.0:1789760794.877196:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784159411106 time 784159150690 state 0x1 ref 1 00000100:00100000:3.0:1789760794.877218:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784159411106 time 784208149211 state 0x0 ref 1 00000100:00100000:3.0:1789760794.877231:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484380800:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760794.877280:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25886720, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:3.0:1789760794.877289:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.877305:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:3.0:1789760794.877310:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760794.877316:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25886720->25890815] (req 25886720->25890815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.877331:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25886720->25890815] (req 25886720->25890815) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760794.912118:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25886720 -> 25890816 00010000:00010000:3.0:1789760794.912137:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25886720->25890815] (req 25886720->25890815) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.912150:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25886720->25890815] (req 25886720->25890815) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.912160:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25886720->25890815] (req 25886720->25890815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.912167:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760794.912172:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25886720->25890815] (req 25886720->25890815) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.912181:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25886720->25890815] (req 25886720->25890815) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.912201:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25886720->25890815] (req 25886720->25890815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.912209:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a68d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25886720->25890815] (req 25886720->25890815) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760794.912299:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484380800:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35065us (35356us total) trans 4294973638 rc 0/0 00000100:00100000:3.0:1789760794.912309:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6349 00000100:00100000:3.0:1789760794.912319:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784159411106 time 784208149211 state 0x2 ref 0 00000001:00080000:3.0:1789760794.912359:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973638 is committed 00000100:00100000:2.0:1789760794.917185:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760794.917403:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484380928 00000100:00100000:0.0:1789760794.917415:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484380928 00000100:00100000:0.0:1789760794.917477:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784159411106 time 784208149211 state 0x0 ref 1 00000100:00100000:0.0:1789760794.917493:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784208409627 time 784208149211 state 0x1 ref 1 00000100:00100000:0.0:1789760794.917518:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784208409627 time 784248449274 state 0x0 ref 1 00000100:00100000:0.0:1789760794.917533:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484380928:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760794.917599:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25890816, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760794.917610:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.917626:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760794.917630:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760794.917637:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25890816->25894911] (req 25890816->25894911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.917653:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25890816->25894911] (req 25890816->25894911) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760794.956394:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25890816 -> 25894912 00010000:00010000:1.0:1789760794.956415:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25890816->25894911] (req 25890816->25894911) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760794.956416:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973639 is committed 00010000:00010000:1.0:1789760794.956441:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25890816->25894911] (req 25890816->25894911) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.956451:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25890816->25894911] (req 25890816->25894911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.956459:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760794.956464:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25890816->25894911] (req 25890816->25894911) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.956473:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25890816->25894911] (req 25890816->25894911) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.956492:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25890816->25894911] (req 25890816->25894911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760794.956500:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000d92ea4b6/0x2cf382639620a694 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25890816->25894911] (req 25890816->25894911) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760794.956593:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484380928:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39059us (39412us total) trans 4294973639 rc 0/0 00000100:00100000:1.0:1789760794.956603:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6350 00000100:00100000:1.0:1789760794.956613:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784208409627 time 784248449274 state 0x2 ref 0 00000100:00100000:1.0:1789760794.963167:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760794.963315:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484381056 00000100:00100000:0.0:1789760794.963325:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484381056 00000100:00100000:0.0:1789760794.963379:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784208409627 time 784248449274 state 0x0 ref 1 00000100:00100000:0.0:1789760794.963394:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784248709690 time 784248449274 state 0x1 ref 1 00000100:00100000:0.0:1789760794.963426:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784248709690 time 784294356577 state 0x0 ref 1 00000100:00100000:0.0:1789760794.963439:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484381056:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760794.963486:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25894912, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:0.0:1789760794.963495:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.963509:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:0.0:1789760794.963513:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760794.963519:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25894912->25899007] (req 25894912->25899007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760794.963532:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25894912->25899007] (req 25894912->25899007) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760794.998460:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973640 is committed 00002000:00010000:3.0:1789760794.998569:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25894912 -> 25899008 00010000:00010000:3.0:1789760794.998583:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25894912->25899007] (req 25894912->25899007) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.998596:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25894912->25899007] (req 25894912->25899007) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.998604:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25894912->25899007] (req 25894912->25899007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.998617:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760794.998621:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25894912->25899007] (req 25894912->25899007) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.998631:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25894912->25899007] (req 25894912->25899007) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.998650:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25894912->25899007] (req 25894912->25899007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760794.998658:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a69b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25894912->25899007] (req 25894912->25899007) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760794.998764:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484381056:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35322us (35601us total) trans 4294973640 rc 0/0 00000100:00100000:3.0:1789760794.998775:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6351 00000100:00100000:3.0:1789760794.998784:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784248709690 time 784294356577 state 0x2 ref 0 00000100:00100000:3.0:1789760795.002175:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760795.002726:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484381184 00000100:00100000:1.0:1789760795.002737:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484381184 00000100:00100000:1.0:1789760795.002791:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784248709690 time 784294356577 state 0x0 ref 1 00000100:00100000:1.0:1789760795.002806:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784294616993 time 784294356577 state 0x1 ref 1 00000100:00100000:1.0:1789760795.002842:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784294616993 time 784333773455 state 0x0 ref 1 00000100:00100000:1.0:1789760795.002855:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f706bc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484381184:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760795.002907:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25899008, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760795.002927:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.002944:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760795.002949:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760795.002955:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25899008->25903103] (req 25899008->25903103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.002982:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25899008->25903103] (req 25899008->25903103) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760795.020557:0:3480:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c9c15180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508945920:0@lo:41:osp-pre-1-0.0 00000100:00100000:1.0:1789760795.020633:0:3480:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760795.020952:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508945920 00000100:00100000:1.0:1789760795.020961:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508945920 00000100:00100000:1.0:1789760795.021006:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 482 00000100:00100000:1.0:1789760795.021015:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508945920:12345-0@lo:41:osp-pre-1-0.0 00000004:00000024:1.0:1789760795.021054:0:6479:0:(mdt_handler.c:540:mdt_statfs()) blocks cached 0 granted 6815744 pending 0 free 2261778432 avail 2259681280 00000100:00100000:1.0:1789760795.021746:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:3480:x1876699508945920:12345-0@lo:41:osp-pre-1-0.0 Request processed in 730us (1113us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760795.021755:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 482 00000100:00100000:2.0:1789760795.021819:0:3480:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c9c15180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3480:1876699508945920:0@lo:41:osp-pre-1-0.0 00000001:00080000:3.0:1789760795.034336:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973641 is committed 00002000:00010000:1.0:1789760795.034474:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25899008 -> 25903104 00010000:00010000:1.0:1789760795.034489:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25899008->25903103] (req 25899008->25903103) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.034502:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25899008->25903103] (req 25899008->25903103) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.034511:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25899008->25903103] (req 25899008->25903103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.034519:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760795.034524:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25899008->25903103] (req 25899008->25903103) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.034533:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25899008->25903103] (req 25899008->25903103) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.034553:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25899008->25903103] (req 25899008->25903103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.034560:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25899008->25903103] (req 25899008->25903103) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760795.034648:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f706bc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484381184:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31791us (32476us total) trans 4294973641 rc 0/0 00000100:00100000:1.0:1789760795.034659:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6352 00000100:00100000:1.0:1789760795.034668:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784294616993 time 784333773455 state 0x2 ref 0 00000100:00100000:1.0:1789760795.047777:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760795.048018:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484381312 00000100:00100000:3.0:1789760795.048028:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484381312 00000100:00100000:3.0:1789760795.048315:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784294616993 time 784333773455 state 0x0 ref 1 00000100:00100000:3.0:1789760795.048330:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784334033871 time 784333773455 state 0x1 ref 1 00000100:00100000:1.0:1789760795.055201:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784334033871 time 784386118842 state 0x0 ref 1 00000100:00100000:1.0:1789760795.055238:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484381312:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760795.055397:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25903104, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760795.055406:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.055423:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760795.055427:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760795.055434:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25903104->25907199] (req 25903104->25907199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.055449:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25903104->25907199] (req 25903104->25907199) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760795.084491:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973642 is committed 00002000:00010000:0.0:1789760795.084570:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25903104 -> 25907200 00010000:00010000:0.0:1789760795.084588:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25903104->25907199] (req 25903104->25907199) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.084602:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25903104->25907199] (req 25903104->25907199) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.084611:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25903104->25907199] (req 25903104->25907199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.084619:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cfa00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760795.084624:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25903104->25907199] (req 25903104->25907199) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.084634:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25903104->25907199] (req 25903104->25907199) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.084655:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25903104->25907199] (req 25903104->25907199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.084663:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cb8a145d/0x2cf382639620a6a9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25903104->25907199] (req 25903104->25907199) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760795.084924:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484381312:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29682us (37140us total) trans 4294973642 rc 0/0 00000100:00100000:0.0:1789760795.084935:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6353 00000100:00100000:0.0:1789760795.084944:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784334033871 time 784386118842 state 0x2 ref 0 00000100:00100000:3.0:1789760795.088673:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760795.088950:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484381440 00000100:00100000:1.0:1789760795.089093:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484381440 00000100:00100000:1.0:1789760795.089381:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784334033871 time 784386118842 state 0x0 ref 1 00000100:00100000:1.0:1789760795.089399:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784386379258 time 784386118842 state 0x1 ref 1 00000100:00100000:1.0:1789760795.089481:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784386379258 time 784420410879 state 0x0 ref 1 00000100:00100000:1.0:1789760795.089494:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484381440:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760795.089544:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25907200, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760795.089553:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.089569:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760795.089574:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760795.089580:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25907200->25911295] (req 25907200->25911295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.089594:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25907200->25911295] (req 25907200->25911295) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760795.148932:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973643 is committed 00002000:00010000:2.0:1789760795.149034:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25907200 -> 25911296 00010000:00010000:2.0:1789760795.149049:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25907200->25911295] (req 25907200->25911295) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.149082:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25907200->25911295] (req 25907200->25911295) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.149094:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25907200->25911295] (req 25907200->25911295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.149102:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cee00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760795.149107:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25907200->25911295] (req 25907200->25911295) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.149122:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25907200->25911295] (req 25907200->25911295) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.149142:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25907200->25911295] (req 25907200->25911295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.149150:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f390894d/0x2cf382639620a6b0 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25907200->25911295] (req 25907200->25911295) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760795.149285:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abaa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484381440:12345-192.168.203.10@tcp:4:dd.0 Request processed in 59788us (60615us total) trans 4294973643 rc 0/0 00000100:00100000:2.0:1789760795.149302:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6354 00000100:00100000:2.0:1789760795.149378:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784386379258 time 784420410879 state 0x2 ref 0 00000100:00100000:3.0:1789760795.155727:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760795.155846:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484381568 00000100:00100000:1.0:1789760795.155857:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484381568 00000100:00100000:1.0:1789760795.155923:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784386379258 time 784420410879 state 0x0 ref 1 00000100:00100000:1.0:1789760795.155940:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784420671295 time 784420410879 state 0x1 ref 1 00000100:00100000:1.0:1789760795.155975:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784420671295 time 784486906104 state 0x0 ref 1 00000100:00100000:1.0:1789760795.155989:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484381568:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760795.156041:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25911296, mtime: 0x6aad951a, atime: 0x0, ctime: 0x6aad951a, blocks: 0xc807 00010000:00010000:1.0:1789760795.156051:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.156108:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760795.156113:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760795.156122:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25911296->25915391] (req 25911296->25915391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.156137:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25911296->25915391] (req 25911296->25915391) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760795.218627:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973644 is committed 00002000:00010000:3.0:1789760795.218699:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25911296 -> 25915392 00002000:00010000:3.0:1789760795.218704:0:8108:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760794 -> 1789760795 00002000:00010000:3.0:1789760795.218707:0:8108:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760794 -> 1789760795 00010000:00010000:3.0:1789760795.218719:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25911296->25915391] (req 25911296->25915391) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.218732:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25911296->25915391] (req 25911296->25915391) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.218742:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25911296->25915391] (req 25911296->25915391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.218749:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760795.218755:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25911296->25915391] (req 25911296->25915391) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.218764:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25911296->25915391] (req 25911296->25915391) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.218783:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25911296->25915391] (req 25911296->25915391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.218790:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000009b1bcae8/0x2cf382639620a6b7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25911296->25915391] (req 25911296->25915391) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760795.218887:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484381568:12345-192.168.203.10@tcp:4:dd.0 Request processed in 62895us (63164us total) trans 4294973644 rc 0/0 00000100:00100000:3.0:1789760795.218898:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6355 00000100:00100000:3.0:1789760795.218908:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784420671295 time 784486906104 state 0x2 ref 0 00000100:00100000:0.0:1789760795.222984:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760795.224506:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484381696 00000100:00100000:2.0:1789760795.224519:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484381696 00000100:00100000:2.0:1789760795.224581:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784420671295 time 784486906104 state 0x0 ref 1 00000100:00100000:2.0:1789760795.224596:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784487166520 time 784486906104 state 0x1 ref 1 00000100:00100000:2.0:1789760795.224631:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784487166520 time 784555562690 state 0x0 ref 1 00000100:00100000:2.0:1789760795.224645:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484381696:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760795.224695:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25915392, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:2.0:1789760795.224704:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.224722:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:2.0:1789760795.224727:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760795.224734:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25915392->25919487] (req 25915392->25919487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.224749:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25915392->25919487] (req 25915392->25919487) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760795.270903:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973645 is committed 00002000:00010000:2.0:1789760795.271017:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25915392 -> 25919488 00010000:00010000:2.0:1789760795.271031:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25915392->25919487] (req 25915392->25919487) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.271045:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25915392->25919487] (req 25915392->25919487) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.271055:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25915392->25919487] (req 25915392->25919487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.271062:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760795.271098:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25915392->25919487] (req 25915392->25919487) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.271108:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25915392->25919487] (req 25915392->25919487) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.271129:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25915392->25919487] (req 25915392->25919487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.271138:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6be lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25915392->25919487] (req 25915392->25919487) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760795.271544:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c810a300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484381696:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46897us (48565us total) trans 4294973645 rc 0/0 00000100:00100000:3.0:1789760795.271555:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6356 00000100:00100000:3.0:1789760795.271565:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784487166520 time 784555562690 state 0x2 ref 0 00000100:00100000:1.0:1789760795.277951:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760795.278389:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484381824 00000100:00100000:2.0:1789760795.278400:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484381824 00000100:00100000:2.0:1789760795.278458:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784487166520 time 784555562690 state 0x0 ref 1 00000100:00100000:2.0:1789760795.278473:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784555823106 time 784555562690 state 0x1 ref 1 00000100:00100000:2.0:1789760795.278547:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784555823106 time 784609478170 state 0x0 ref 1 00000100:00100000:2.0:1789760795.278560:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484381824:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760795.278613:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25919488, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:2.0:1789760795.278622:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.278638:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:2.0:1789760795.278643:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760795.278649:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25919488->25923583] (req 25919488->25923583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.278662:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25919488->25923583] (req 25919488->25923583) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760795.328494:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25919488 -> 25923584 00010000:00010000:0.0:1789760795.328515:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25919488->25923583] (req 25919488->25923583) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.328528:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25919488->25923583] (req 25919488->25923583) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.328537:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25919488->25923583] (req 25919488->25923583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.328544:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760795.328549:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25919488->25923583] (req 25919488->25923583) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.328558:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25919488->25923583] (req 25919488->25923583) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.328577:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25919488->25923583] (req 25919488->25923583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.328585:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007c456088/0x2cf382639620a6c5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25919488->25923583] (req 25919488->25923583) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760795.328677:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484381824:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50115us (50731us total) trans 4294973646 rc 0/0 00000100:00100000:0.0:1789760795.328688:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6357 00000100:00100000:0.0:1789760795.328706:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784555823106 time 784609478170 state 0x2 ref 0 00000001:00080000:3.0:1789760795.329245:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973646 is committed 00000100:00100000:0.0:1789760795.338681:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760795.339033:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484381952 00000100:00100000:1.0:1789760795.339043:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484381952 00000100:00100000:1.0:1789760795.339187:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784555823106 time 784609478170 state 0x0 ref 1 00000100:00100000:1.0:1789760795.339203:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784609738586 time 784609478170 state 0x1 ref 1 00000100:00100000:1.0:1789760795.339292:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784609738586 time 784670221533 state 0x0 ref 1 00000100:00100000:1.0:1789760795.339306:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484381952:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760795.339355:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25923584, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:1.0:1789760795.339365:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.339381:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760795.339386:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760795.339392:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25923584->25927679] (req 25923584->25927679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.339405:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25923584->25927679] (req 25923584->25927679) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760795.381026:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973647 is committed 00002000:00010000:1.0:1789760795.381182:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25923584 -> 25927680 00010000:00010000:1.0:1789760795.381195:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25923584->25927679] (req 25923584->25927679) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.381205:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25923584->25927679] (req 25923584->25927679) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.381215:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25923584->25927679] (req 25923584->25927679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.381227:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760795.381232:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25923584->25927679] (req 25923584->25927679) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.381242:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25923584->25927679] (req 25923584->25927679) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.381263:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25923584->25927679] (req 25923584->25927679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.381272:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6cc lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25923584->25927679] (req 25923584->25927679) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760795.381402:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484381952:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42094us (42726us total) trans 4294973647 rc 0/0 00000100:00100000:1.0:1789760795.381411:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6358 00000100:00100000:1.0:1789760795.381431:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784609738586 time 784670221533 state 0x2 ref 0 00000100:00100000:1.0:1789760795.386621:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760795.386782:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484382080 00000100:00100000:3.0:1789760795.386790:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484382080 00000100:00100000:3.0:1789760795.386852:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784609738586 time 784670221533 state 0x0 ref 1 00000100:00100000:3.0:1789760795.386870:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784670481949 time 784670221533 state 0x1 ref 1 00000100:00100000:3.0:1789760795.386901:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784670481949 time 784717832605 state 0x0 ref 1 00000100:00100000:3.0:1789760795.386923:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484382080:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760795.386985:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25927680, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:3.0:1789760795.386995:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.387016:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:3.0:1789760795.387020:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760795.387027:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25927680->25931775] (req 25927680->25931775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.387040:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25927680->25931775] (req 25927680->25931775) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760795.429861:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973648 is committed 00002000:00010000:1.0:1789760795.429959:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25927680 -> 25931776 00010000:00010000:1.0:1789760795.429974:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25927680->25931775] (req 25927680->25931775) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.429988:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25927680->25931775] (req 25927680->25931775) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.430086:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25927680->25931775] (req 25927680->25931775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.430096:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760795.430101:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25927680->25931775] (req 25927680->25931775) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.430110:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25927680->25931775] (req 25927680->25931775) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.430130:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25927680->25931775] (req 25927680->25931775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.430138:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6d3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25927680->25931775] (req 25927680->25931775) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760795.430235:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484382080:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43318us (43617us total) trans 4294973648 rc 0/0 00000100:00100000:1.0:1789760795.430245:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6359 00000100:00100000:1.0:1789760795.430256:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784670481949 time 784717832605 state 0x2 ref 0 00000100:00100000:3.0:1789760795.444302:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760795.444644:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484382208 00000100:00100000:1.0:1789760795.444654:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484382208 00000100:00100000:1.0:1789760795.444718:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784670481949 time 784717832605 state 0x0 ref 1 00000100:00100000:1.0:1789760795.444733:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784718093021 time 784717832605 state 0x1 ref 1 00000100:00100000:1.0:1789760795.444803:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784718093021 time 784775733502 state 0x0 ref 1 00000100:00100000:1.0:1789760795.444817:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484382208:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760795.444863:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25931776, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:1.0:1789760795.444872:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.444886:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:1.0:1789760795.444890:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760795.444896:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25931776->25935871] (req 25931776->25935871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.444909:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25931776->25935871] (req 25931776->25935871) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760795.492818:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973649 is committed 00002000:00010000:1.0:1789760795.493100:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25931776 -> 25935872 00010000:00010000:1.0:1789760795.493135:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25931776->25935871] (req 25931776->25935871) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.493160:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25931776->25935871] (req 25931776->25935871) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.493173:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25931776->25935871] (req 25931776->25935871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.493180:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760795.493185:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25931776->25935871] (req 25931776->25935871) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.493194:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25931776->25935871] (req 25931776->25935871) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.493283:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25931776->25935871] (req 25931776->25935871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.493294:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000087e6f88d/0x2cf382639620a6da lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25931776->25935871] (req 25931776->25935871) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760795.493450:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8abbb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484382208:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48631us (49152us total) trans 4294973649 rc 0/0 00000100:00100000:1.0:1789760795.493462:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6360 00000100:00100000:1.0:1789760795.493471:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784718093021 time 784775733502 state 0x2 ref 0 00000100:00100000:3.0:1789760795.496942:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760795.497137:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484382336 00000100:00100000:0.0:1789760795.497152:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484382336 00000100:00100000:0.0:1789760795.497242:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784718093021 time 784775733502 state 0x0 ref 1 00000100:00100000:0.0:1789760795.497261:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784775993918 time 784775733502 state 0x1 ref 1 00000100:00100000:0.0:1789760795.497291:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784775993918 time 784828222329 state 0x0 ref 1 00000100:00100000:0.0:1789760795.497308:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484382336:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760795.497375:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25935872, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:0.0:1789760795.497386:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.497406:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:0.0:1789760795.497411:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760795.497418:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25935872->25939967] (req 25935872->25939967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.497435:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25935872->25939967] (req 25935872->25939967) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.535963:0:6462:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:2.0:1789760795.569513:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973650 is committed 00002000:00010000:3.0:1789760795.570219:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25935872 -> 25939968 00010000:00010000:3.0:1789760795.570243:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25935872->25939967] (req 25935872->25939967) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.570260:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25935872->25939967] (req 25935872->25939967) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.570272:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25935872->25939967] (req 25935872->25939967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.570280:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760795.570287:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25935872->25939967] (req 25935872->25939967) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.570297:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25935872->25939967] (req 25935872->25939967) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.570323:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25935872->25939967] (req 25935872->25939967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.570332:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a6e1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25935872->25939967] (req 25935872->25939967) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760795.570510:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab9500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484382336:12345-192.168.203.10@tcp:4:dd.0 Request processed in 73200us (73577us total) trans 4294973650 rc 0/0 00000100:00100000:3.0:1789760795.570522:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6361 00000100:00100000:3.0:1789760795.570534:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784775993918 time 784828222329 state 0x2 ref 0 00000100:00100000:0.0:1789760795.573616:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760795.573758:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484382464 00000100:00100000:2.0:1789760795.573772:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484382464 00000100:00100000:2.0:1789760795.573849:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784775993918 time 784828222329 state 0x0 ref 1 00000100:00100000:2.0:1789760795.573868:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784828482745 time 784828222329 state 0x1 ref 1 00000100:00100000:2.0:1789760795.574013:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784828482745 time 784904942214 state 0x0 ref 1 00000100:00100000:2.0:1789760795.574029:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484382464:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760795.574635:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25939968, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:0.0:1789760795.574649:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.574670:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:0.0:1789760795.574676:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760795.574684:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25939968->25944063] (req 25939968->25944063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.574701:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25939968->25944063] (req 25939968->25944063) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760795.651487:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973651 is committed 00002000:00010000:3.0:1789760795.651634:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25939968 -> 25944064 00010000:00010000:3.0:1789760795.651655:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25939968->25944063] (req 25939968->25944063) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.651673:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25939968->25944063] (req 25939968->25944063) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.651684:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25939968->25944063] (req 25939968->25944063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.651707:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760795.651713:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25939968->25944063] (req 25939968->25944063) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.651729:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25939968->25944063] (req 25939968->25944063) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.651755:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25939968->25944063] (req 25939968->25944063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.651774:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a6e8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25939968->25944063] (req 25939968->25944063) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760795.652270:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484382464:12345-192.168.203.10@tcp:4:dd.0 Request processed in 78235us (78661us total) trans 4294973651 rc 0/0 00000100:00100000:3.0:1789760795.652283:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6362 00000100:00100000:3.0:1789760795.652294:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784828482745 time 784904942214 state 0x2 ref 0 00000100:00100000:1.0:1789760795.665188:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760795.666191:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484382592 00000100:00100000:3.0:1789760795.666207:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484382592 00000100:00100000:3.0:1789760795.666285:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784828482745 time 784904942214 state 0x0 ref 1 00000100:00100000:3.0:1789760795.666303:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784905202630 time 784904942214 state 0x1 ref 1 00000100:00100000:3.0:1789760795.666392:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784905202630 time 784997321391 state 0x0 ref 1 00000100:00100000:3.0:1789760795.666408:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484382592:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760795.666478:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25944064, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:3.0:1789760795.666488:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.666508:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=784s v=(0 0) 00010000:00010000:3.0:1789760795.666513:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760795.666520:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25944064->25948159] (req 25944064->25948159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.666536:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25944064->25948159] (req 25944064->25948159) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760795.707739:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25944064 -> 25948160 00010000:00010000:0.0:1789760795.707769:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25944064->25948159] (req 25944064->25948159) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.707786:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25944064->25948159] (req 25944064->25948159) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.707796:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25944064->25948159] (req 25944064->25948159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.707804:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760795.707811:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25944064->25948159] (req 25944064->25948159) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.707822:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25944064->25948159] (req 25944064->25948159) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.707851:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25944064->25948159] (req 25944064->25948159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.707859:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006af685a1/0x2cf382639620a6ef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25944064->25948159] (req 25944064->25948159) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760795.707995:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484382592:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41583us (42817us total) trans 4294973652 rc 0/0 00000100:00100000:0.0:1789760795.708016:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6363 00000100:00100000:0.0:1789760795.708030:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784905202630 time 784997321391 state 0x2 ref 0 00000001:00080000:2.0:1789760795.708241:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973652 is committed 00000100:00100000:2.0:1789760795.711285:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760795.711528:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484382720 00000100:00100000:0.0:1789760795.711555:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484382720 00000100:00100000:0.0:1789760795.711639:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784905202630 time 784997321391 state 0x0 ref 1 00000100:00100000:0.0:1789760795.711659:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784997581807 time 784997321391 state 0x1 ref 1 00000100:00100000:0.0:1789760795.711705:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784997581807 time 785042634878 state 0x0 ref 1 00000100:00100000:0.0:1789760795.711721:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484382720:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760795.711787:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25948160, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:0.0:1789760795.711798:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.711818:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:0.0:1789760795.711823:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760795.711831:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25948160->25952255] (req 25948160->25952255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.711848:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25948160->25952255] (req 25948160->25952255) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760795.748184:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25948160 -> 25952256 00010000:00010000:2.0:1789760795.748213:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25948160->25952255] (req 25948160->25952255) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.748229:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25948160->25952255] (req 25948160->25952255) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.748239:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25948160->25952255] (req 25948160->25952255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.748247:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760795.748253:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25948160->25952255] (req 25948160->25952255) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.748269:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25948160->25952255] (req 25948160->25952255) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.748295:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25948160->25952255] (req 25948160->25952255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.748302:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a6f6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25948160->25952255] (req 25948160->25952255) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760795.748484:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0df80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484382720:12345-192.168.203.10@tcp:4:dd.0 Request processed in 36758us (37204us total) trans 4294973653 rc 0/0 00000100:00100000:2.0:1789760795.748497:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6364 00000100:00100000:2.0:1789760795.748509:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784997581807 time 785042634878 state 0x2 ref 0 00000001:00080000:3.0:1789760795.748605:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973653 is committed 00000100:00100000:0.0:1789760795.752820:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760795.752920:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484382848 00000100:00100000:3.0:1789760795.752933:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484382848 00000100:00100000:3.0:1789760795.753062:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 784997581807 time 785042634878 state 0x0 ref 1 00000100:00100000:3.0:1789760795.753148:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785042895294 time 785042634878 state 0x1 ref 1 00000100:00100000:3.0:1789760795.753209:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785042895294 time 785084139248 state 0x0 ref 1 00000100:00100000:3.0:1789760795.753223:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484382848:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760795.753284:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25952256, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:3.0:1789760795.753293:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.753313:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:3.0:1789760795.753318:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760795.753325:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25952256->25956351] (req 25952256->25956351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.753343:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25952256->25956351] (req 25952256->25956351) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760795.780634:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973654 is committed 00002000:00010000:2.0:1789760795.781254:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25952256 -> 25956352 00010000:00010000:2.0:1789760795.781277:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25952256->25956351] (req 25952256->25956351) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.781292:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25952256->25956351] (req 25952256->25956351) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.781301:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25952256->25956351] (req 25952256->25956351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.781307:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cce00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760795.781312:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25952256->25956351] (req 25952256->25956351) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.781321:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25952256->25956351] (req 25952256->25956351) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.781347:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25952256->25956351] (req 25952256->25956351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.781354:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c01f5ba9/0x2cf382639620a6fd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25952256->25956351] (req 25952256->25956351) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760795.781519:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8108a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484382848:12345-192.168.203.10@tcp:4:dd.0 Request processed in 28291us (28704us total) trans 4294973654 rc 0/0 00000100:00100000:2.0:1789760795.781532:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6365 00000100:00100000:2.0:1789760795.781545:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785042895294 time 785084139248 state 0x2 ref 0 00000100:00100000:0.0:1789760795.785259:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760795.785442:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484382976 00000100:00100000:3.0:1789760795.785456:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484382976 00000100:00100000:3.0:1789760795.785534:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785042895294 time 785084139248 state 0x0 ref 1 00000100:00100000:3.0:1789760795.785553:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785084399664 time 785084139248 state 0x1 ref 1 00000100:00100000:3.0:1789760795.785590:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785084399664 time 785116520321 state 0x0 ref 1 00000100:00100000:3.0:1789760795.785605:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484382976:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760795.785670:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25956352, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:3.0:1789760795.785681:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.785702:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:3.0:1789760795.785707:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760795.785714:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25956352->25960447] (req 25956352->25960447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760795.785731:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25956352->25960447] (req 25956352->25960447) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760795.842783:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25956352 -> 25960448 00010000:00010000:0.0:1789760795.842818:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25956352->25960447] (req 25956352->25960447) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.842834:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25956352->25960447] (req 25956352->25960447) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.842845:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25956352->25960447] (req 25956352->25960447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.842853:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cde00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760795.842860:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25956352->25960447] (req 25956352->25960447) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.842871:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25956352->25960447] (req 25956352->25960447) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.842895:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25956352->25960447] (req 25956352->25960447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.842903:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000076cb6105/0x2cf382639620a704 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25956352->25960447] (req 25956352->25960447) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760795.843032:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c8109500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484382976:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57424us (57780us total) trans 4294973655 rc 0/0 00000100:00100000:0.0:1789760795.843044:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6366 00000100:00100000:0.0:1789760795.843057:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785084399664 time 785116520321 state 0x2 ref 0 00000001:00080000:2.0:1789760795.843432:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973655 is committed 00000100:00100000:3.0:1789760795.856487:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760795.857878:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484383104 00000100:00100000:0.0:1789760795.857893:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484383104 00000100:00100000:1.0:1789760795.858168:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785084399664 time 785116520321 state 0x0 ref 1 00000100:00100000:1.0:1789760795.858190:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785116780737 time 785116520321 state 0x1 ref 1 00000100:00100000:1.0:1789760795.858278:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785116780737 time 785189207831 state 0x0 ref 1 00000100:00100000:1.0:1789760795.858295:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484383104:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760795.858365:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25960448, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:1.0:1789760795.858377:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.858399:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:1.0:1789760795.858404:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760795.858412:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25960448->25964543] (req 25960448->25964543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760795.858429:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25960448->25964543] (req 25960448->25964543) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760795.905845:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973656 is committed 00002000:00010000:2.0:1789760795.906151:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25960448 -> 25964544 00010000:00010000:2.0:1789760795.906171:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25960448->25964543] (req 25960448->25964543) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.906187:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25960448->25964543] (req 25960448->25964543) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.906198:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25960448->25964543] (req 25960448->25964543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.906210:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760795.906216:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25960448->25964543] (req 25960448->25964543) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.906226:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25960448->25964543] (req 25960448->25964543) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.906250:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25960448->25964543] (req 25960448->25964543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.906258:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a70b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25960448->25964543] (req 25960448->25964543) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760795.906425:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484383104:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48127us (49946us total) trans 4294973656 rc 0/0 00000100:00100000:2.0:1789760795.906437:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6367 00000100:00100000:2.0:1789760795.906449:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785116780737 time 785189207831 state 0x2 ref 0 00000100:00100000:1.0:1789760795.910221:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760795.910342:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484383232 00000100:00100000:2.0:1789760795.910356:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484383232 00000100:00100000:2.0:1789760795.910428:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785116780737 time 785189207831 state 0x0 ref 1 00000100:00100000:2.0:1789760795.910448:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785189468247 time 785189207831 state 0x1 ref 1 00000100:00100000:2.0:1789760795.910475:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785189468247 time 785241405016 state 0x0 ref 1 00000100:00100000:2.0:1789760795.910492:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484383232:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760795.910560:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25964544, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:2.0:1789760795.910572:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.910593:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:2.0:1789760795.910598:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760795.910606:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25964544->25968639] (req 25964544->25968639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.910622:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25964544->25968639] (req 25964544->25968639) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760795.960119:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973657 is committed 00002000:00010000:0.0:1789760795.960250:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25964544 -> 25968640 00010000:00010000:0.0:1789760795.960270:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25964544->25968639] (req 25964544->25968639) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.960286:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25964544->25968639] (req 25964544->25968639) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.960297:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25964544->25968639] (req 25964544->25968639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.960306:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760795.960311:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25964544->25968639] (req 25964544->25968639) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.960322:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25964544->25968639] (req 25964544->25968639) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.960348:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25964544->25968639] (req 25964544->25968639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760795.960357:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a712 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25964544->25968639] (req 25964544->25968639) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760795.960484:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484383232:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49988us (50272us total) trans 4294973657 rc 0/0 00000100:00100000:0.0:1789760795.960497:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6368 00000100:00100000:0.0:1789760795.960510:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785189468247 time 785241405016 state 0x2 ref 0 00000100:00100000:0.0:1789760795.965275:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760795.965611:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484383360 00000100:00100000:2.0:1789760795.965625:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484383360 00000100:00100000:2.0:1789760795.965699:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785189468247 time 785241405016 state 0x0 ref 1 00000100:00100000:2.0:1789760795.965717:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785241665432 time 785241405016 state 0x1 ref 1 00000100:00100000:2.0:1789760795.965752:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785241665432 time 785296682619 state 0x0 ref 1 00000100:00100000:2.0:1789760795.965765:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b80000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484383360:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760795.966210:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25968640, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:2.0:1789760795.966221:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.966241:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:2.0:1789760795.966246:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760795.966253:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25968640->25972735] (req 25968640->25972735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760795.966269:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25968640->25972735] (req 25968640->25972735) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760796.021472:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973658 is committed 00002000:00010000:0.0:1789760796.021904:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25968640 -> 25972736 00010000:00010000:0.0:1789760796.021932:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25968640->25972735] (req 25968640->25972735) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.021955:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25968640->25972735] (req 25968640->25972735) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.022008:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25968640->25972735] (req 25968640->25972735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.022018:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760796.022024:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25968640->25972735] (req 25968640->25972735) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.022035:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25968640->25972735] (req 25968640->25972735) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.022060:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25968640->25972735] (req 25968640->25972735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.022092:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a719 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25968640->25972735] (req 25968640->25972735) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760796.022946:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b80000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484383360:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57175us (57677us total) trans 4294973658 rc 0/0 00000100:00100000:0.0:1789760796.022961:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6369 00000100:00100000:0.0:1789760796.023267:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785241665432 time 785296682619 state 0x2 ref 0 00000100:00100000:2.0:1789760796.038140:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760796.038191:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484383488 00000100:00100000:0.0:1789760796.038204:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484383488 00000100:00100000:0.0:1789760796.038292:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785241665432 time 785296682619 state 0x0 ref 1 00000100:00100000:0.0:1789760796.038310:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785296943035 time 785296682619 state 0x1 ref 1 00000100:00100000:0.0:1789760796.038337:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785296943035 time 785369268525 state 0x0 ref 1 00000100:00100000:0.0:1789760796.038358:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484383488:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760796.038416:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25972736, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:0.0:1789760796.038425:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.038444:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:0.0:1789760796.038449:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760796.038456:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25972736->25976831] (req 25972736->25976831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.038472:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25972736->25976831] (req 25972736->25976831) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1789760796.039355:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:3.0:1789760796.099354:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973659 is committed 00002000:00010000:0.0:1789760796.099479:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25972736 -> 25976832 00010000:00010000:0.0:1789760796.099499:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25972736->25976831] (req 25972736->25976831) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.099514:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25972736->25976831] (req 25972736->25976831) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.099525:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25972736->25976831] (req 25972736->25976831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.099534:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760796.099548:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25972736->25976831] (req 25972736->25976831) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.099558:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25972736->25976831] (req 25972736->25976831) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.099584:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25972736->25976831] (req 25972736->25976831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.099592:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a720 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25972736->25976831] (req 25972736->25976831) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760796.100176:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484383488:12345-192.168.203.10@tcp:4:dd.0 Request processed in 61818us (62041us total) trans 4294973659 rc 0/0 00000100:00100000:0.0:1789760796.100190:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6370 00000100:00100000:0.0:1789760796.100202:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785296943035 time 785369268525 state 0x2 ref 0 00000100:00100000:1.0:1789760796.104974:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760796.105281:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484383616 00000100:00100000:2.0:1789760796.105295:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484383616 00000100:00100000:2.0:1789760796.105373:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785296943035 time 785369268525 state 0x0 ref 1 00000100:00100000:2.0:1789760796.105392:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785369528941 time 785369268525 state 0x1 ref 1 00000100:00100000:2.0:1789760796.105426:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785369528941 time 785436356204 state 0x0 ref 1 00000100:00100000:2.0:1789760796.105440:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484383616:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760796.105509:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25976832, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:2.0:1789760796.105520:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.105542:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:2.0:1789760796.105547:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760796.105553:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25976832->25980927] (req 25976832->25980927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.105571:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25976832->25980927] (req 25976832->25980927) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760796.148570:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973660 is committed 00002000:00010000:3.0:1789760796.148657:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25976832 -> 25980928 00010000:00010000:3.0:1789760796.148677:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25976832->25980927] (req 25976832->25980927) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.148704:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25976832->25980927] (req 25976832->25980927) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.148714:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25976832->25980927] (req 25976832->25980927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.148721:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760796.148726:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25976832->25980927] (req 25976832->25980927) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.148735:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25976832->25980927] (req 25976832->25980927) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.148759:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25976832->25980927] (req 25976832->25980927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.148766:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a727 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25976832->25980927] (req 25976832->25980927) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760796.148889:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484383616:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43444us (43989us total) trans 4294973660 rc 0/0 00000100:00100000:3.0:1789760796.148917:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6371 00000100:00100000:3.0:1789760796.148932:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785369528941 time 785436356204 state 0x2 ref 0 00000100:00100000:2.0:1789760796.152540:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760796.152616:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484383744 00000100:00100000:0.0:1789760796.152630:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484383744 00000100:00100000:0.0:1789760796.152714:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785369528941 time 785436356204 state 0x0 ref 1 00000100:00100000:0.0:1789760796.152733:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785436616620 time 785436356204 state 0x1 ref 1 00000100:00100000:0.0:1789760796.152773:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785436616620 time 785483702656 state 0x0 ref 1 00000100:00100000:0.0:1789760796.152788:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484383744:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760796.153028:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25980928, mtime: 0x6aad951b, atime: 0x0, ctime: 0x6aad951b, blocks: 0xc807 00010000:00010000:0.0:1789760796.153040:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.153059:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:0.0:1789760796.153169:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760796.153179:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25980928->25985023] (req 25980928->25985023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.153198:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25980928->25985023] (req 25980928->25985023) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760796.215660:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973661 is committed 00002000:00010000:2.0:1789760796.216114:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25980928 -> 25985024 00002000:00010000:2.0:1789760796.216120:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760795 -> 1789760796 00002000:00010000:2.0:1789760796.216122:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760795 -> 1789760796 00010000:00010000:2.0:1789760796.216140:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25980928->25985023] (req 25980928->25985023) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.216157:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25980928->25985023] (req 25980928->25985023) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.216167:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25980928->25985023] (req 25980928->25985023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.216176:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760796.216182:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25980928->25985023] (req 25980928->25985023) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.216192:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25980928->25985023] (req 25980928->25985023) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.216219:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25980928->25985023] (req 25980928->25985023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.216235:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a72e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25980928->25985023] (req 25980928->25985023) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760796.216371:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484383744:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63580us (63838us total) trans 4294973661 rc 0/0 00000100:00100000:2.0:1789760796.216383:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6372 00000100:00100000:2.0:1789760796.216396:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785436616620 time 785483702656 state 0x2 ref 0 00000100:00100000:2.0:1789760796.232043:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760796.232157:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484383872 00000100:00100000:0.0:1789760796.232173:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484383872 00000100:00100000:0.0:1789760796.232255:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785436616620 time 785483702656 state 0x0 ref 1 00000100:00100000:0.0:1789760796.232271:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785483963072 time 785483702656 state 0x1 ref 1 00000100:00100000:0.0:1789760796.232305:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785483963072 time 785563237029 state 0x0 ref 1 00000100:00100000:0.0:1789760796.232316:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484383872:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760796.232377:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25985024, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:0.0:1789760796.232389:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.232405:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:0.0:1789760796.232410:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760796.232417:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25985024->25989119] (req 25985024->25989119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.232435:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25985024->25989119] (req 25985024->25989119) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760796.280122:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25985024 -> 25989120 00010000:00010000:2.0:1789760796.280153:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25985024->25989119] (req 25985024->25989119) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.280170:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25985024->25989119] (req 25985024->25989119) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.280181:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25985024->25989119] (req 25985024->25989119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.280190:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760796.280196:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25985024->25989119] (req 25985024->25989119) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.280207:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25985024->25989119] (req 25985024->25989119) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.280239:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25985024->25989119] (req 25985024->25989119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.280248:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a735 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25985024->25989119] (req 25985024->25989119) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760796.280390:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484383872:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48067us (48358us total) trans 4294973662 rc 0/0 00000100:00100000:2.0:1789760796.280404:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6373 00000100:00100000:2.0:1789760796.280418:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785483963072 time 785563237029 state 0x2 ref 0 00000001:00080000:1.0:1789760796.280516:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973662 is committed 00000100:00100000:1.0:1789760796.284720:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760796.284830:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484384000 00000100:00100000:2.0:1789760796.284844:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484384000 00000100:00100000:2.0:1789760796.284934:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785483963072 time 785563237029 state 0x0 ref 1 00000100:00100000:2.0:1789760796.284953:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785563497445 time 785563237029 state 0x1 ref 1 00000100:00100000:2.0:1789760796.285032:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785563497445 time 785615962422 state 0x0 ref 1 00000100:00100000:2.0:1789760796.285049:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484384000:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760796.285158:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25989120, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:2.0:1789760796.285169:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.285188:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:2.0:1789760796.285193:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760796.285202:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25989120->25993215] (req 25989120->25993215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.285218:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25989120->25993215] (req 25989120->25993215) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760796.353886:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973663 is committed 00002000:00010000:2.0:1789760796.354128:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25989120 -> 25993216 00010000:00010000:2.0:1789760796.354148:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25989120->25993215] (req 25989120->25993215) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.354165:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25989120->25993215] (req 25989120->25993215) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.354177:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25989120->25993215] (req 25989120->25993215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.354186:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760796.354192:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25989120->25993215] (req 25989120->25993215) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.354203:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25989120->25993215] (req 25989120->25993215) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.354229:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25989120->25993215] (req 25989120->25993215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.354242:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000f13dfa98/0x2cf382639620a73c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25989120->25993215] (req 25989120->25993215) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760796.354363:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c17100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484384000:12345-192.168.203.10@tcp:4:dd.0 Request processed in 69311us (69653us total) trans 4294973663 rc 0/0 00000100:00100000:2.0:1789760796.354376:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6374 00000100:00100000:2.0:1789760796.354390:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785563497445 time 785615962422 state 0x2 ref 0 00000100:00100000:2.0:1789760796.360545:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760796.360656:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484384128 00000100:00100000:3.0:1789760796.360670:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484384128 00000100:00100000:3.0:1789760796.360915:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785563497445 time 785615962422 state 0x0 ref 1 00000100:00100000:3.0:1789760796.360936:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785616222838 time 785615962422 state 0x1 ref 1 00000100:00100000:3.0:1789760796.361612:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785616222838 time 785692537166 state 0x0 ref 1 00000100:00100000:3.0:1789760796.361633:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484384128:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760796.361772:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25993216, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:3.0:1789760796.361784:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.361806:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:3.0:1789760796.361812:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760796.361820:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25993216->25997311] (req 25993216->25997311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.361836:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25993216->25997311] (req 25993216->25997311) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760796.390612:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973664 is committed 00002000:00010000:3.0:1789760796.390892:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25993216 -> 25997312 00010000:00010000:3.0:1789760796.390924:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25993216->25997311] (req 25993216->25997311) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.390941:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25993216->25997311] (req 25993216->25997311) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.390952:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25993216->25997311] (req 25993216->25997311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.390960:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760796.390966:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25993216->25997311] (req 25993216->25997311) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.390977:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25993216->25997311] (req 25993216->25997311) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.391004:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25993216->25997311] (req 25993216->25997311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.391014:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a743 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25993216->25997311] (req 25993216->25997311) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760796.391194:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484384128:12345-192.168.203.10@tcp:4:dd.0 Request processed in 29559us (30655us total) trans 4294973664 rc 0/0 00000100:00100000:3.0:1789760796.391211:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6375 00000100:00100000:3.0:1789760796.391224:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785616222838 time 785692537166 state 0x2 ref 0 00000100:00100000:3.0:1789760796.398307:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760796.400425:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484384256 00000100:00100000:1.0:1789760796.400456:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484384256 00000100:00100000:1.0:1789760796.400549:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785616222838 time 785692537166 state 0x0 ref 1 00000100:00100000:1.0:1789760796.400568:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785692797582 time 785692537166 state 0x1 ref 1 00000100:00100000:1.0:1789760796.400603:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785692797582 time 785731533148 state 0x0 ref 1 00000100:00100000:1.0:1789760796.400619:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484384256:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760796.400944:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 25997312, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:2.0:1789760796.400956:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.401041:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:2.0:1789760796.401047:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760796.401055:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25997312->26001407] (req 25997312->26001407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.401111:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25997312->26001407] (req 25997312->26001407) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760796.459162:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 25997312 -> 26001408 00010000:00010000:2.0:1789760796.459192:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25997312->26001407] (req 25997312->26001407) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.459213:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25997312->26001407] (req 25997312->26001407) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.459222:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25997312->26001407] (req 25997312->26001407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.459229:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760796.459235:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25997312->26001407] (req 25997312->26001407) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.459244:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25997312->26001407] (req 25997312->26001407) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.459270:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25997312->26001407] (req 25997312->26001407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.459279:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a74a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [25997312->26001407] (req 25997312->26001407) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760796.459412:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8aba300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484384256:12345-192.168.203.10@tcp:4:dd.0 Request processed in 58789us (61111us total) trans 4294973665 rc 0/0 00000100:00100000:2.0:1789760796.459427:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6376 00000100:00100000:2.0:1789760796.459441:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785692797582 time 785731533148 state 0x2 ref 0 00000001:00080000:0.0:1789760796.459559:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973665 is committed 00000100:00100000:2.0:1789760796.466489:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760796.466639:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484384384 00000100:00100000:0.0:1789760796.466654:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484384384 00000100:00100000:0.0:1789760796.466742:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785692797582 time 785731533148 state 0x0 ref 1 00000100:00100000:0.0:1789760796.466959:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785731793564 time 785731533148 state 0x1 ref 1 00000100:00100000:0.0:1789760796.467002:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785731793564 time 785797932040 state 0x0 ref 1 00000100:00100000:0.0:1789760796.467021:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484384384:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760796.467119:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26001408, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:0.0:1789760796.467131:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.467222:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:0.0:1789760796.467229:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760796.467236:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26001408->26005503] (req 26001408->26005503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.467255:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26001408->26005503] (req 26001408->26005503) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760796.527461:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973666 is committed 00002000:00010000:2.0:1789760796.527543:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26001408 -> 26005504 00010000:00010000:2.0:1789760796.527564:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26001408->26005503] (req 26001408->26005503) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.527580:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26001408->26005503] (req 26001408->26005503) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.527590:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26001408->26005503] (req 26001408->26005503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.527598:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760796.527604:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26001408->26005503] (req 26001408->26005503) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.527614:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26001408->26005503] (req 26001408->26005503) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.527640:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26001408->26005503] (req 26001408->26005503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.527649:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000e2c94c6b/0x2cf382639620a751 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26001408->26005503] (req 26001408->26005503) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760796.527777:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484384384:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60755us (61297us total) trans 4294973666 rc 0/0 00000100:00100000:2.0:1789760796.527790:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6377 00000100:00100000:2.0:1789760796.527802:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785731793564 time 785797932040 state 0x2 ref 0 00000100:00100000:3.0:1789760796.530202:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760796.530424:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484384512 00000100:00100000:2.0:1789760796.530438:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484384512 00000100:00100000:2.0:1789760796.530515:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785731793564 time 785797932040 state 0x0 ref 1 00000100:00100000:2.0:1789760796.530534:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785798192456 time 785797932040 state 0x1 ref 1 00000100:00100000:2.0:1789760796.530641:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785798192456 time 785861570102 state 0x0 ref 1 00000100:00100000:2.0:1789760796.530657:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484384512:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760796.530732:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26005504, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:2.0:1789760796.530743:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.530763:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:2.0:1789760796.530768:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760796.530776:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26005504->26009599] (req 26005504->26009599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.530792:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26005504->26009599] (req 26005504->26009599) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.556733:0:8119:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:0.0:1789760796.583549:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26005504 -> 26009600 00010000:00010000:0.0:1789760796.583579:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26005504->26009599] (req 26005504->26009599) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.583595:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26005504->26009599] (req 26005504->26009599) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.583606:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26005504->26009599] (req 26005504->26009599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.583614:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79e00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760796.583621:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26005504->26009599] (req 26005504->26009599) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.583632:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26005504->26009599] (req 26005504->26009599) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.583657:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26005504->26009599] (req 26005504->26009599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.583665:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000041baf81/0x2cf382639620a758 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26005504->26009599] (req 26005504->26009599) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760796.583802:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484384512:12345-192.168.203.10@tcp:4:dd.0 Request processed in 53141us (53608us total) trans 4294973667 rc 0/0 00000100:00100000:0.0:1789760796.583816:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6378 00000100:00100000:0.0:1789760796.583829:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785798192456 time 785861570102 state 0x2 ref 0 00000001:00080000:3.0:1789760796.584004:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973667 is committed 00000100:00100000:3.0:1789760796.588011:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760796.588345:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484384640 00000100:00100000:0.0:1789760796.588359:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484384640 00000100:00100000:0.0:1789760796.588445:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785798192456 time 785861570102 state 0x0 ref 1 00000100:00100000:0.0:1789760796.588466:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785861830518 time 785861570102 state 0x1 ref 1 00000100:00100000:0.0:1789760796.588542:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785861830518 time 785919471023 state 0x0 ref 1 00000100:00100000:0.0:1789760796.588559:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484384640:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760796.588630:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26009600, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:0.0:1789760796.588640:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.588659:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:0.0:1789760796.588664:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760796.588672:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26009600->26013695] (req 26009600->26013695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.588796:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26009600->26013695] (req 26009600->26013695) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760796.638008:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26009600 -> 26013696 00010000:00010000:3.0:1789760796.638035:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26009600->26013695] (req 26009600->26013695) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.638051:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26009600->26013695] (req 26009600->26013695) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.638103:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26009600->26013695] (req 26009600->26013695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.638112:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760796.638118:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26009600->26013695] (req 26009600->26013695) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.638127:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26009600->26013695] (req 26009600->26013695) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.638153:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26009600->26013695] (req 26009600->26013695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.638163:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000aa52b999/0x2cf382639620a75f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26009600->26013695] (req 26009600->26013695) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760796.638252:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973668 is committed 00000100:00100000:3.0:1789760796.638288:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8ab8380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484384640:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49726us (50286us total) trans 4294973668 rc 0/0 00000100:00100000:3.0:1789760796.638302:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6379 00000100:00100000:3.0:1789760796.638315:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785861830518 time 785919471023 state 0x2 ref 0 00000100:00100000:2.0:1789760796.641543:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760796.641659:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484384768 00000100:00100000:0.0:1789760796.641671:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484384768 00000100:00100000:0.0:1789760796.641749:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785861830518 time 785919471023 state 0x0 ref 1 00000100:00100000:0.0:1789760796.641768:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785919731439 time 785919471023 state 0x1 ref 1 00000100:00100000:0.0:1789760796.641806:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785919731439 time 785972736203 state 0x0 ref 1 00000100:00100000:0.0:1789760796.641820:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484384768:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760796.641884:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26013696, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:0.0:1789760796.641894:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.641925:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=785s v=(0 0) 00010000:00010000:0.0:1789760796.641931:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760796.641939:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26013696->26017791] (req 26013696->26017791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.641970:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26013696->26017791] (req 26013696->26017791) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760796.701677:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973669 is committed 00002000:00010000:2.0:1789760796.701864:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26013696 -> 26017792 00010000:00010000:2.0:1789760796.701885:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26013696->26017791] (req 26013696->26017791) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.701918:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26013696->26017791] (req 26013696->26017791) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.701997:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26013696->26017791] (req 26013696->26017791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.702062:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760796.702148:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26013696->26017791] (req 26013696->26017791) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.702161:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26013696->26017791] (req 26013696->26017791) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.702190:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26013696->26017791] (req 26013696->26017791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.702198:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000ed376468/0x2cf382639620a766 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26013696->26017791] (req 26013696->26017791) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760796.702483:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484384768:12345-192.168.203.10@tcp:4:dd.0 Request processed in 60659us (60947us total) trans 4294973669 rc 0/0 00000100:00100000:2.0:1789760796.702497:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6380 00000100:00100000:2.0:1789760796.702509:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785919731439 time 785972736203 state 0x2 ref 0 00000100:00100000:0.0:1789760796.706131:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760796.706244:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484384896 00000100:00100000:2.0:1789760796.706261:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484384896 00000100:00100000:2.0:1789760796.706351:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785919731439 time 785972736203 state 0x0 ref 1 00000100:00100000:2.0:1789760796.706371:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785972996619 time 785972736203 state 0x1 ref 1 00000100:00100000:2.0:1789760796.706514:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785972996619 time 786037443114 state 0x0 ref 1 00000100:00100000:2.0:1789760796.706588:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484384896:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760796.706667:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26017792, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:2.0:1789760796.706678:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.706698:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:2.0:1789760796.706704:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760796.706712:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26017792->26021887] (req 26017792->26021887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.706731:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26017792->26021887] (req 26017792->26021887) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760796.748504:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973670 is committed 00002000:00010000:2.0:1789760796.748643:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26017792 -> 26021888 00010000:00010000:2.0:1789760796.748663:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26017792->26021887] (req 26017792->26021887) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.748679:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26017792->26021887] (req 26017792->26021887) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.748763:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26017792->26021887] (req 26017792->26021887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.748770:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760796.748776:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26017792->26021887] (req 26017792->26021887) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.748786:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26017792->26021887] (req 26017792->26021887) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.748811:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26017792->26021887] (req 26017792->26021887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.748820:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a76d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26017792->26021887] (req 26017792->26021887) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760796.749213:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf501b83800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484384896:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42621us (43088us total) trans 4294973670 rc 0/0 00000100:00100000:2.0:1789760796.749229:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6381 00000100:00100000:2.0:1789760796.749243:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785972996619 time 786037443114 state 0x2 ref 0 00000100:00100000:2.0:1789760796.754049:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760796.754285:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484385024 00000100:00100000:0.0:1789760796.754299:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484385024 00000100:00100000:2.0:1789760796.754555:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 785972996619 time 786037443114 state 0x0 ref 1 00000100:00100000:2.0:1789760796.754575:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786037703530 time 786037443114 state 0x1 ref 1 00000100:00100000:2.0:1789760796.754649:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786037703530 time 786085578505 state 0x0 ref 1 00000100:00100000:2.0:1789760796.754666:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484385024:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760796.754745:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26021888, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:2.0:1789760796.754757:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.754779:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:2.0:1789760796.754784:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760796.754792:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26021888->26025983] (req 26021888->26025983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.754811:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26021888->26025983] (req 26021888->26025983) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760796.812157:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973671 is committed 00002000:00010000:2.0:1789760796.812336:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26021888 -> 26025984 00010000:00010000:2.0:1789760796.812357:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26021888->26025983] (req 26021888->26025983) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.812373:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26021888->26025983] (req 26021888->26025983) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.812395:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26021888->26025983] (req 26021888->26025983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.812404:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760796.812411:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26021888->26025983] (req 26021888->26025983) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.812422:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26021888->26025983] (req 26021888->26025983) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.812449:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26021888->26025983] (req 26021888->26025983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.812458:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a4618ca0/0x2cf382639620a774 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26021888->26025983] (req 26021888->26025983) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760796.812580:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0dc00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484385024:12345-192.168.203.10@tcp:4:dd.0 Request processed in 57910us (58688us total) trans 4294973671 rc 0/0 00000100:00100000:2.0:1789760796.812593:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6382 00000100:00100000:2.0:1789760796.812605:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786037703530 time 786085578505 state 0x2 ref 0 00000100:00100000:1.0:1789760796.815810:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760796.815931:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484385152 00000100:00100000:3.0:1789760796.815945:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484385152 00000100:00100000:3.0:1789760796.816039:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786037703530 time 786085578505 state 0x0 ref 1 00000100:00100000:3.0:1789760796.816061:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786085838921 time 786085578505 state 0x1 ref 1 00000100:00100000:3.0:1789760796.816141:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786085838921 time 786147070185 state 0x0 ref 1 00000100:00100000:3.0:1789760796.816157:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484385152:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760796.816230:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26025984, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:3.0:1789760796.816242:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.816263:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:3.0:1789760796.816268:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760796.816276:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26025984->26030079] (req 26025984->26030079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.816293:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26025984->26030079] (req 26025984->26030079) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760796.858188:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973672 is committed 00002000:00010000:3.0:1789760796.858505:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26025984 -> 26030080 00010000:00010000:3.0:1789760796.858527:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26025984->26030079] (req 26025984->26030079) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.858542:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26025984->26030079] (req 26025984->26030079) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.858553:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26025984->26030079] (req 26025984->26030079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.858562:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760796.858568:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26025984->26030079] (req 26025984->26030079) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.858579:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26025984->26030079] (req 26025984->26030079) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.858606:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26025984->26030079] (req 26025984->26030079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760796.858615:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a77b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26025984->26030079] (req 26025984->26030079) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760796.858808:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c14a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484385152:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42647us (43117us total) trans 4294973672 rc 0/0 00000100:00100000:3.0:1789760796.858821:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6383 00000100:00100000:3.0:1789760796.858833:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786085838921 time 786147070185 state 0x2 ref 0 00000100:00100000:2.0:1789760796.862233:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760796.862314:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484385280 00000100:00100000:1.0:1789760796.862330:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484385280 00000100:00100000:1.0:1789760796.862408:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786085838921 time 786147070185 state 0x0 ref 1 00000100:00100000:1.0:1789760796.862429:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786147330601 time 786147070185 state 0x1 ref 1 00000100:00100000:1.0:1789760796.862539:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786147330601 time 786193468374 state 0x0 ref 1 00000100:00100000:1.0:1789760796.862555:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484385280:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760796.862621:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26030080, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:1.0:1789760796.862633:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760796.862653:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:1.0:1789760796.862658:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760796.862665:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26030080->26034175] (req 26030080->26034175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760796.862682:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26030080->26034175] (req 26030080->26034175) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760796.910282:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973673 is committed 00002000:00010000:0.0:1789760796.910433:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26030080 -> 26034176 00010000:00010000:0.0:1789760796.910454:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26030080->26034175] (req 26030080->26034175) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.910470:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26030080->26034175] (req 26030080->26034175) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.910481:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26030080->26034175] (req 26030080->26034175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.910489:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760796.910495:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26030080->26034175] (req 26030080->26034175) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.910506:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26030080->26034175] (req 26030080->26034175) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.910534:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26030080->26034175] (req 26030080->26034175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.910543:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a782 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26030080->26034175] (req 26030080->26034175) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760796.910861:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0e300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484385280:12345-192.168.203.10@tcp:4:dd.0 Request processed in 48303us (48636us total) trans 4294973673 rc 0/0 00000100:00100000:0.0:1789760796.910874:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6384 00000100:00100000:0.0:1789760796.910887:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786147330601 time 786193468374 state 0x2 ref 0 00000100:00100000:2.0:1789760796.913624:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760796.913881:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484385408 00000100:00100000:0.0:1789760796.913894:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484385408 00000100:00100000:0.0:1789760796.914184:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786147330601 time 786193468374 state 0x0 ref 1 00000100:00100000:0.0:1789760796.914203:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786193728790 time 786193468374 state 0x1 ref 1 00000100:00100000:1.0:1789760796.914339:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786193728790 time 786245266881 state 0x0 ref 1 00000100:00100000:1.0:1789760796.914356:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484385408:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760796.914420:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26034176, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:1.0:1789760796.914430:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760796.914456:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:1.0:1789760796.914461:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760796.914468:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26034176->26038271] (req 26034176->26038271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760796.914484:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26034176->26038271] (req 26034176->26038271) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760796.956546:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973674 is committed 00002000:00010000:0.0:1789760796.956693:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26034176 -> 26038272 00010000:00010000:0.0:1789760796.956714:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26034176->26038271] (req 26034176->26038271) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.956737:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26034176->26038271] (req 26034176->26038271) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.956748:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26034176->26038271] (req 26034176->26038271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.956757:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760796.956763:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26034176->26038271] (req 26034176->26038271) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.956773:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26034176->26038271] (req 26034176->26038271) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.956796:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26034176->26038271] (req 26034176->26038271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760796.956805:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000062c8e907/0x2cf382639620a789 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26034176->26038271] (req 26034176->26038271) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760796.956913:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0f480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484385408:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42547us (43287us total) trans 4294973674 rc 0/0 00000100:00100000:0.0:1789760796.956924:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6385 00000100:00100000:0.0:1789760796.956944:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786193728790 time 786245266881 state 0x2 ref 0 00000100:00100000:1.0:1789760796.960540:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760796.960895:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484385536 00000100:00100000:2.0:1789760796.960917:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484385536 00000100:00100000:2.0:1789760796.961435:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786193728790 time 786245266881 state 0x0 ref 1 00000100:00100000:2.0:1789760796.961453:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786245527297 time 786245266881 state 0x1 ref 1 00000100:00100000:2.0:1789760796.961490:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786245527297 time 786292421398 state 0x0 ref 1 00000100:00100000:2.0:1789760796.961510:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c16a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484385536:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760796.961575:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26038272, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:2.0:1789760796.961587:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.961608:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:2.0:1789760796.961613:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760796.961621:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26038272->26042367] (req 26038272->26042367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760796.961638:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26038272->26042367] (req 26038272->26042367) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760797.036885:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973675 is committed 00002000:00010000:3.0:1789760797.037026:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26038272 -> 26042368 00010000:00010000:3.0:1789760797.037049:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26038272->26042367] (req 26038272->26042367) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.037082:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26038272->26042367] (req 26038272->26042367) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.037094:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26038272->26042367] (req 26038272->26042367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.037103:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760797.037109:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26038272->26042367] (req 26038272->26042367) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.037120:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26038272->26042367] (req 26038272->26042367) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.037149:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26038272->26042367] (req 26038272->26042367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.037162:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001bdc2bb4/0x2cf382639620a790 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26038272->26042367] (req 26038272->26042367) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760797.037294:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c16a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484385536:12345-192.168.203.10@tcp:4:dd.0 Request processed in 75781us (76762us total) trans 4294973675 rc 0/0 00000100:00100000:3.0:1789760797.037306:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6386 00000100:00100000:3.0:1789760797.037319:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786245527297 time 786292421398 state 0x2 ref 0 00000100:00100000:2.0:1789760797.045316:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760797.045593:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484385664 00000100:00100000:3.0:1789760797.045606:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484385664 00000100:00100000:3.0:1789760797.045690:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786245527297 time 786292421398 state 0x0 ref 1 00000100:00100000:3.0:1789760797.045710:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786292681814 time 786292421398 state 0x1 ref 1 00000100:00100000:3.0:1789760797.045749:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786292681814 time 786376679557 state 0x0 ref 1 00000100:00100000:3.0:1789760797.045766:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484385664:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760797.045833:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26042368, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:3.0:1789760797.045845:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.045864:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:3.0:1789760797.045868:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760797.045875:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26042368->26046463] (req 26042368->26046463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.045896:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26042368->26046463] (req 26042368->26046463) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760797.112660:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26042368 -> 26046464 00010000:00010000:0.0:1789760797.112694:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26042368->26046463] (req 26042368->26046463) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.112712:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26042368->26046463] (req 26042368->26046463) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.112723:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26042368->26046463] (req 26042368->26046463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.112730:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cca00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760797.112735:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26042368->26046463] (req 26042368->26046463) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.112746:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26042368->26046463] (req 26042368->26046463) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.112777:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26042368->26046463] (req 26042368->26046463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.112786:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000da3694f8/0x2cf382639620a797 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26042368->26046463] (req 26042368->26046463) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760797.112917:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0c000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484385664:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67148us (67610us total) trans 4294973676 rc 0/0 00000100:00100000:0.0:1789760797.112935:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6387 00000100:00100000:0.0:1789760797.112946:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786292681814 time 786376679557 state 0x2 ref 0 00000001:00080000:2.0:1789760797.114940:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973676 is committed 00000100:00100000:3.0:1789760797.119321:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760797.119531:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484385792 00000100:00100000:0.0:1789760797.119544:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484385792 00000100:00100000:0.0:1789760797.119622:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786292681814 time 786376679557 state 0x0 ref 1 00000100:00100000:0.0:1789760797.119640:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786376939973 time 786376679557 state 0x1 ref 1 00000100:00100000:0.0:1789760797.119716:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786376939973 time 786450646525 state 0x0 ref 1 00000100:00100000:0.0:1789760797.119733:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdd880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484385792:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760797.119802:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26046464, mtime: 0x6aad951c, atime: 0x0, ctime: 0x6aad951c, blocks: 0xc807 00010000:00010000:0.0:1789760797.119813:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.119833:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:0.0:1789760797.119839:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760797.119847:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26046464->26050559] (req 26046464->26050559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.120103:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26046464->26050559] (req 26046464->26050559) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:0.0:1789760797.170258:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26046464 -> 26050560 00002000:00010000:0.0:1789760797.170270:0:8107:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760796 -> 1789760797 00002000:00010000:0.0:1789760797.170273:0:8107:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760796 -> 1789760797 00010000:00010000:0.0:1789760797.170290:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26046464->26050559] (req 26046464->26050559) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.170306:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26046464->26050559] (req 26046464->26050559) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.170318:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26046464->26050559] (req 26046464->26050559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.170326:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760797.170332:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26046464->26050559] (req 26046464->26050559) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.170342:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26046464->26050559] (req 26046464->26050559) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.170367:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26046464->26050559] (req 26046464->26050559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.170377:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a79e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26046464->26050559] (req 26046464->26050559) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760797.170499:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdd880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484385792:12345-192.168.203.10@tcp:4:dd.0 Request processed in 50763us (51185us total) trans 4294973677 rc 0/0 00000100:00100000:0.0:1789760797.170512:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6388 00000100:00100000:0.0:1789760797.170525:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786376939973 time 786450646525 state 0x2 ref 0 00000001:00080000:0.0:1789760797.170577:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973677 is committed 00000100:00100000:1.0:1789760797.177192:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760797.177411:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484385920 00000100:00100000:2.0:1789760797.177426:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484385920 00000100:00100000:2.0:1789760797.177504:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786376939973 time 786450646525 state 0x0 ref 1 00000100:00100000:2.0:1789760797.177524:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786450906941 time 786450646525 state 0x1 ref 1 00000100:00100000:2.0:1789760797.177588:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786450906941 time 786508518479 state 0x0 ref 1 00000100:00100000:2.0:1789760797.177603:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484385920:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760797.177671:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26050560, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:2.0:1789760797.177682:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.177702:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:2.0:1789760797.177708:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760797.177716:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26050560->26054655] (req 26050560->26054655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.177733:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26050560->26054655] (req 26050560->26054655) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760797.218424:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26050560 -> 26054656 00010000:00010000:2.0:1789760797.218454:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26050560->26054655] (req 26050560->26054655) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760797.218479:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973678 is committed 00010000:00010000:2.0:1789760797.218482:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26050560->26054655] (req 26050560->26054655) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.218495:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26050560->26054655] (req 26050560->26054655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.218505:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760797.218511:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26050560->26054655] (req 26050560->26054655) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.218521:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26050560->26054655] (req 26050560->26054655) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.218552:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26050560->26054655] (req 26050560->26054655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.218562:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000006289698c/0x2cf382639620a7a5 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26050560->26054655] (req 26050560->26054655) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760797.218679:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484385920:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41074us (41502us total) trans 4294973678 rc 0/0 00000100:00100000:2.0:1789760797.218692:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6389 00000100:00100000:2.0:1789760797.218705:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786450906941 time 786508518479 state 0x2 ref 0 00000100:00100000:0.0:1789760797.222622:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760797.223693:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484386048 00000100:00100000:3.0:1789760797.223711:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484386048 00000100:00100000:3.0:1789760797.223871:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786450906941 time 786508518479 state 0x0 ref 1 00000100:00100000:3.0:1789760797.223893:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786508778895 time 786508518479 state 0x1 ref 1 00000100:00100000:3.0:1789760797.223941:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786508778895 time 786554870935 state 0x0 ref 1 00000100:00100000:3.0:1789760797.223956:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484386048:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760797.224037:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26054656, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:3.0:1789760797.224050:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.224096:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:3.0:1789760797.224101:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760797.224109:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26054656->26058751] (req 26054656->26058751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.224127:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26054656->26058751] (req 26054656->26058751) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760797.270618:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26054656 -> 26058752 00010000:00010000:2.0:1789760797.270645:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26054656->26058751] (req 26054656->26058751) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.270661:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26054656->26058751] (req 26054656->26058751) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.270671:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26054656->26058751] (req 26054656->26058751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.270679:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc800 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760797.270752:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26054656->26058751] (req 26054656->26058751) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.270772:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26054656->26058751] (req 26054656->26058751) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.270808:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26054656->26058751] (req 26054656->26058751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.270818:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000011ffbda8/0x2cf382639620a7ac lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26054656->26058751] (req 26054656->26058751) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760797.270953:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb9f3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2423:x1876699484386048:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46993us (48338us total) trans 4294973679 rc 0/0 00000100:00100000:2.0:1789760797.270965:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6390 00000100:00100000:2.0:1789760797.270978:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786508778895 time 786554870935 state 0x2 ref 0 00000001:00080000:1.0:1789760797.272425:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973679 is committed 00000100:00100000:3.0:1789760797.276100:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760797.276435:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484386176 00000100:00100000:0.0:1789760797.276451:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484386176 00000100:00100000:0.0:1789760797.276995:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786508778895 time 786554870935 state 0x0 ref 1 00000100:00100000:0.0:1789760797.277105:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786555131351 time 786554870935 state 0x1 ref 1 00000100:00100000:0.0:1789760797.277136:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786555131351 time 786608066952 state 0x0 ref 1 00000100:00100000:0.0:1789760797.277154:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdf100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484386176:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760797.277223:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26058752, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:0.0:1789760797.277235:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.277255:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:0.0:1789760797.277260:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760797.277268:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26058752->26062847] (req 26058752->26062847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.277285:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26058752->26062847] (req 26058752->26062847) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:1.0:1789760797.372402:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26058752 -> 26062848 00010000:00010000:1.0:1789760797.372431:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26058752->26062847] (req 26058752->26062847) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.372445:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26058752->26062847] (req 26058752->26062847) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.372454:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26058752->26062847] (req 26058752->26062847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.372462:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760797.372467:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26058752->26062847] (req 26058752->26062847) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.372483:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26058752->26062847] (req 26058752->26062847) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.372509:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26058752->26062847] (req 26058752->26062847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.372518:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000425bf0ea/0x2cf382639620a7b3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26058752->26062847] (req 26058752->26062847) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760797.372657:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdf100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484386176:12345-192.168.203.10@tcp:4:dd.0 Request processed in 95502us (97664us total) trans 4294973680 rc 0/0 00000100:00100000:1.0:1789760797.372670:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6391 00000100:00100000:1.0:1789760797.372825:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786555131351 time 786608066952 state 0x2 ref 0 00000001:00080000:0.0:1789760797.373388:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973680 is committed 00000100:00100000:2.0:1789760797.376445:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760797.377116:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484386304 00000100:00100000:0.0:1789760797.377131:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484386304 00000100:00100000:0.0:1789760797.377218:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786555131351 time 786608066952 state 0x0 ref 1 00000100:00100000:0.0:1789760797.377237:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786608327368 time 786608066952 state 0x1 ref 1 00000100:00100000:0.0:1789760797.377262:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786608327368 time 786708193356 state 0x0 ref 1 00000100:00100000:0.0:1789760797.377278:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484386304:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760797.377341:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26062848, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:0.0:1789760797.377352:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.377373:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:0.0:1789760797.377378:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760797.377385:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26062848->26066943] (req 26062848->26066943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.377402:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26062848->26066943] (req 26062848->26066943) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760797.444323:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973681 is committed 00002000:00010000:2.0:1789760797.444855:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26062848 -> 26066944 00010000:00010000:2.0:1789760797.444875:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26062848->26066943] (req 26062848->26066943) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.444906:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26062848->26066943] (req 26062848->26066943) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.444915:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26062848->26066943] (req 26062848->26066943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.444928:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb198600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760797.444940:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26062848->26066943] (req 26062848->26066943) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.444951:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26062848->26066943] (req 26062848->26066943) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.444978:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26062848->26066943] (req 26062848->26066943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.444985:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000004e2818d/0x2cf382639620a7ba lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26062848->26066943] (req 26062848->26066943) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760797.445129:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4dfb0d500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484386304:12345-192.168.203.10@tcp:4:dd.0 Request processed in 67850us (68696us total) trans 4294973681 rc 0/0 00000100:00100000:2.0:1789760797.445141:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6392 00000100:00100000:2.0:1789760797.445152:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786608327368 time 786708193356 state 0x2 ref 0 00000100:00100000:2.0:1789760797.452468:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760797.452587:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484386432 00000100:00100000:0.0:1789760797.452597:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484386432 00000100:00100000:0.0:1789760797.452658:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786608327368 time 786708193356 state 0x0 ref 1 00000100:00100000:0.0:1789760797.452674:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786708453772 time 786708193356 state 0x1 ref 1 00000100:00100000:0.0:1789760797.452702:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786708453772 time 786783632506 state 0x0 ref 1 00000100:00100000:0.0:1789760797.452718:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb726680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484386432:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760797.452774:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26066944, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:0.0:1789760797.452784:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.452799:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:0.0:1789760797.452803:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760797.452810:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26066944->26071039] (req 26066944->26071039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.452824:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26066944->26071039] (req 26066944->26071039) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760797.517778:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973682 is committed 00002000:00010000:0.0:1789760797.517920:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26066944 -> 26071040 00010000:00010000:0.0:1789760797.517942:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26066944->26071039] (req 26066944->26071039) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.517962:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26066944->26071039] (req 26066944->26071039) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.517971:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26066944->26071039] (req 26066944->26071039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.517980:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760797.517985:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26066944->26071039] (req 26066944->26071039) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.517997:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26066944->26071039] (req 26066944->26071039) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.518021:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26066944->26071039] (req 26066944->26071039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.518029:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7c1 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26066944->26071039] (req 26066944->26071039) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760797.518267:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb726680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484386432:12345-192.168.203.10@tcp:4:dd.0 Request processed in 65547us (65802us total) trans 4294973682 rc 0/0 00000100:00100000:0.0:1789760797.518278:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6393 00000100:00100000:0.0:1789760797.518299:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786708453772 time 786783632506 state 0x2 ref 0 00000100:00100000:3.0:1789760797.528973:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760797.529121:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484386560 00000100:00100000:2.0:1789760797.529136:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484386560 00000100:00100000:2.0:1789760797.529222:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786708453772 time 786783632506 state 0x0 ref 1 00000100:00100000:2.0:1789760797.529242:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786783892922 time 786783632506 state 0x1 ref 1 00000100:00100000:2.0:1789760797.529351:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786783892922 time 786860280425 state 0x0 ref 1 00000100:00100000:2.0:1789760797.529369:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484386560:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760797.529437:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26071040, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:2.0:1789760797.529448:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.529468:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:2.0:1789760797.529473:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760797.529480:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26071040->26075135] (req 26071040->26075135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760797.529498:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26071040->26075135] (req 26071040->26075135) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760797.569487:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973683 is committed 00002000:00010000:0.0:1789760797.569947:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26071040 -> 26075136 00010000:00010000:0.0:1789760797.569967:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26071040->26075135] (req 26071040->26075135) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.569981:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26071040->26075135] (req 26071040->26075135) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.570048:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26071040->26075135] (req 26071040->26075135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.570058:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79c00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760797.570141:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26071040->26075135] (req 26071040->26075135) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.570154:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26071040->26075135] (req 26071040->26075135) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.570180:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26071040->26075135] (req 26071040->26075135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.570189:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000007e9b659a/0x2cf382639620a7c8 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26071040->26075135] (req 26071040->26075135) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760797.570363:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdc000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484386560:12345-192.168.203.10@tcp:4:dd.0 Request processed in 40990us (41399us total) trans 4294973683 rc 0/0 00000100:00100000:0.0:1789760797.570376:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6394 00000100:00100000:0.0:1789760797.570389:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786783892922 time 786860280425 state 0x2 ref 0 00000100:00100000:2.0:1789760797.573790:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760797.573946:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484386688 00000100:00100000:1.0:1789760797.573960:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484386688 00000100:00100000:1.0:1789760797.574042:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786783892922 time 786860280425 state 0x0 ref 1 00000100:00100000:1.0:1789760797.574060:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786860540841 time 786860280425 state 0x1 ref 1 00000100:00100000:1.0:1789760797.574127:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786860540841 time 786905057124 state 0x0 ref 1 00000100:00100000:1.0:1789760797.574142:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb727b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484386688:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760797.574209:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26075136, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:1.0:1789760797.574222:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.574240:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:1.0:1789760797.574246:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760797.574253:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26075136->26079231] (req 26075136->26079231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.574270:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26075136->26079231] (req 26075136->26079231) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760797.583600:0:6461:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00002000:00010000:0.0:1789760797.626205:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26075136 -> 26079232 00010000:00010000:0.0:1789760797.626233:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26075136->26079231] (req 26075136->26079231) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.626249:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26075136->26079231] (req 26075136->26079231) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.626259:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26075136->26079231] (req 26075136->26079231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.626268:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760797.626273:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26075136->26079231] (req 26075136->26079231) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.626285:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26075136->26079231] (req 26075136->26079231) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.626323:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26075136->26079231] (req 26075136->26079231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.626333:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000000579691b/0x2cf382639620a7cf lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26075136->26079231] (req 26075136->26079231) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760797.626469:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb727b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484386688:12345-192.168.203.10@tcp:4:dd.0 Request processed in 52323us (52686us total) trans 4294973684 rc 0/0 00000100:00100000:0.0:1789760797.626483:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6395 00000100:00100000:0.0:1789760797.626496:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786860540841 time 786905057124 state 0x2 ref 0 00000001:00080000:2.0:1789760797.628525:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973684 is committed 00000100:00100000:2.0:1789760797.633549:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760797.633646:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484386816 00000100:00100000:0.0:1789760797.633663:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484386816 00000100:00100000:0.0:1789760797.633754:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786860540841 time 786905057124 state 0x0 ref 1 00000100:00100000:0.0:1789760797.633774:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786905317540 time 786905057124 state 0x1 ref 1 00000100:00100000:0.0:1789760797.633828:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786905317540 time 786964750306 state 0x0 ref 1 00000100:00100000:0.0:1789760797.633843:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484386816:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760797.633921:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26079232, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:0.0:1789760797.633931:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.633951:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=786s v=(0 0) 00010000:00010000:0.0:1789760797.633956:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760797.633963:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26079232->26083327] (req 26079232->26083327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.633979:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26079232->26083327] (req 26079232->26083327) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760797.705965:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973685 is committed 00002000:00010000:0.0:1789760797.706047:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26079232 -> 26083328 00010000:00010000:0.0:1789760797.706085:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26079232->26083327] (req 26079232->26083327) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.706107:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26079232->26083327] (req 26079232->26083327) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.706117:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26079232->26083327] (req 26079232->26083327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.706125:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760797.706130:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26079232->26083327] (req 26079232->26083327) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.706140:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26079232->26083327] (req 26079232->26083327) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.706180:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26079232->26083327] (req 26079232->26083327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.706190:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000066825740/0x2cf382639620a7d6 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26079232->26083327] (req 26079232->26083327) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760797.706328:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb725180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484386816:12345-192.168.203.10@tcp:4:dd.0 Request processed in 72481us (72786us total) trans 4294973685 rc 0/0 00000100:00100000:0.0:1789760797.706341:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6396 00000100:00100000:0.0:1789760797.706352:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786905317540 time 786964750306 state 0x2 ref 0 00000100:00100000:2.0:1789760797.712601:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760797.712946:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484386944 00000100:00100000:0.0:1789760797.712961:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484386944 00000100:00100000:0.0:1789760797.713050:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786905317540 time 786964750306 state 0x0 ref 1 00000100:00100000:0.0:1789760797.713107:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786965010722 time 786964750306 state 0x1 ref 1 00000100:00100000:0.0:1789760797.713136:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786965010722 time 787044067032 state 0x0 ref 1 00000100:00100000:0.0:1789760797.713153:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484386944:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760797.713222:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26083328, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:0.0:1789760797.713234:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.713256:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:0.0:1789760797.713261:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760797.713270:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26083328->26087423] (req 26083328->26087423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.713290:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26083328->26087423] (req 26083328->26087423) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760797.791953:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973686 is committed 00002000:00010000:0.0:1789760797.792151:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26083328 -> 26087424 00010000:00010000:0.0:1789760797.792174:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26083328->26087423] (req 26083328->26087423) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.792192:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26083328->26087423] (req 26083328->26087423) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.792208:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26083328->26087423] (req 26083328->26087423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.792217:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760797.792223:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26083328->26087423] (req 26083328->26087423) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.792234:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26083328->26087423] (req 26083328->26087423) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.792261:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26083328->26087423] (req 26083328->26087423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.792269:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7dd lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26083328->26087423] (req 26083328->26087423) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760797.792422:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4fb724000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484386944:12345-192.168.203.10@tcp:4:dd.0 Request processed in 79265us (79827us total) trans 4294973686 rc 0/0 00000100:00100000:0.0:1789760797.792438:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6397 00000100:00100000:0.0:1789760797.793597:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786965010722 time 787044067032 state 0x2 ref 0 00000100:00100000:2.0:1789760797.808650:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760797.809029:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484387072 00000100:00100000:0.0:1789760797.809043:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484387072 00000100:00100000:0.0:1789760797.809193:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 786965010722 time 787044067032 state 0x0 ref 1 00000100:00100000:0.0:1789760797.809213:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787044327448 time 787044067032 state 0x1 ref 1 00000100:00100000:0.0:1789760797.809248:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787044327448 time 787140178751 state 0x0 ref 1 00000100:00100000:0.0:1789760797.809265:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484387072:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760797.809333:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26087424, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:0.0:1789760797.809344:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.809364:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:0.0:1789760797.809369:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760797.809377:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26087424->26091519] (req 26087424->26091519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.809394:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26087424->26091519] (req 26087424->26091519) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760797.872467:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973687 is committed 00002000:00010000:0.0:1789760797.872572:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26087424 -> 26091520 00010000:00010000:0.0:1789760797.872595:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26087424->26091519] (req 26087424->26091519) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.872611:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26087424->26091519] (req 26087424->26091519) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.872621:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26087424->26091519] (req 26087424->26091519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.872628:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760797.872634:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26087424->26091519] (req 26087424->26091519) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.872645:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26087424->26091519] (req 26087424->26091519) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.872669:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26087424->26091519] (req 26087424->26091519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760797.872677:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7e4 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26087424->26091519] (req 26087424->26091519) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760797.872806:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484387072:12345-192.168.203.10@tcp:4:dd.0 Request processed in 63536us (64161us total) trans 4294973687 rc 0/0 00000100:00100000:0.0:1789760797.872819:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6398 00000100:00100000:0.0:1789760797.872832:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787044327448 time 787140178751 state 0x2 ref 0 00000100:00100000:3.0:1789760797.875522:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760797.875641:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484387200 00000100:00100000:1.0:1789760797.875655:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484387200 00000100:00100000:1.0:1789760797.875728:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787044327448 time 787140178751 state 0x0 ref 1 00000100:00100000:1.0:1789760797.875807:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787140439167 time 787140178751 state 0x1 ref 1 00000100:00100000:1.0:1789760797.875840:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787140439167 time 787206771154 state 0x0 ref 1 00000100:00100000:1.0:1789760797.875854:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484387200:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760797.875925:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26091520, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:1.0:1789760797.875936:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.875955:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:1.0:1789760797.875961:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760797.875969:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26091520->26095615] (req 26091520->26095615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760797.875984:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26091520->26095615] (req 26091520->26095615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:2.0:1789760798.029628:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973688 is committed 00002000:00010000:0.0:1789760798.033796:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26091520 -> 26095616 00010000:00010000:0.0:1789760798.033825:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26091520->26095615] (req 26091520->26095615) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.033840:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26091520->26095615] (req 26091520->26095615) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.033849:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26091520->26095615] (req 26091520->26095615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.033862:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cea00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760798.033868:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26091520->26095615] (req 26091520->26095615) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.033878:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26091520->26095615] (req 26091520->26095615) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.033915:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26091520->26095615] (req 26091520->26095615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.033923:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000a6b47520/0x2cf382639620a7eb lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26091520->26095615] (req 26091520->26095615) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760798.034166:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4defdea00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484387200:12345-192.168.203.10@tcp:4:dd.0 Request processed in 158305us (158649us total) trans 4294973688 rc 0/0 00000100:00100000:0.0:1789760798.034180:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6399 00000100:00100000:0.0:1789760798.034193:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787140439167 time 787206771154 state 0x2 ref 0 00000100:00100000:2.0:1789760798.041791:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760798.042860:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484387328 00000100:00100000:0.0:1789760798.042880:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484387328 00000100:00100000:0.0:1789760798.043053:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787140439167 time 787206771154 state 0x0 ref 1 00000100:00100000:0.0:1789760798.043089:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787207031570 time 787206771154 state 0x1 ref 1 00000100:00100000:0.0:1789760798.043126:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787207031570 time 787374057254 state 0x0 ref 1 00000100:00100000:0.0:1789760798.043146:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484387328:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760798.043211:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26095616, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:0.0:1789760798.043222:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.043242:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:0.0:1789760798.043247:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760798.043255:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26095616->26099711] (req 26095616->26099711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.043271:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26095616->26099711] (req 26095616->26099711) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1789760798.046713:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000001:00080000:2.0:1789760798.089334:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973689 is committed 00002000:00010000:3.0:1789760798.089533:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26095616 -> 26099712 00010000:00010000:3.0:1789760798.089556:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26095616->26099711] (req 26095616->26099711) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.089572:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26095616->26099711] (req 26095616->26099711) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.089582:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26095616->26099711] (req 26095616->26099711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.089590:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760798.089596:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26095616->26099711] (req 26095616->26099711) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.089606:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26095616->26099711] (req 26095616->26099711) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.089633:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26095616->26099711] (req 26095616->26099711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.089641:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000002b2b78ab/0x2cf382639620a7f2 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26095616->26099711] (req 26095616->26099711) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760798.089972:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484387328:12345-192.168.203.10@tcp:4:dd.0 Request processed in 46823us (48299us total) trans 4294973689 rc 0/0 00000100:00100000:3.0:1789760798.089986:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6400 00000100:00100000:3.0:1789760798.089999:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787207031570 time 787374057254 state 0x2 ref 0 00000100:00100000:3.0:1789760798.092635:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb9f0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508946176:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.092754:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760798.092832:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb9f2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508946688:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.092864:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760798.092890:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508947200:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.092931:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760798.092952:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c4579880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508947712:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.092976:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760798.093020:0:3478:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:3.0:1789760798.093028:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb9f1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508946304:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093057:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760798.093110:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb9f3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508946816:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093134:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760798.093156:0:3478:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508947328:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093179:0:3478:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760798.093224:0:12534:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508946688 00000100:00100000:3.0:1789760798.093235:0:12534:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508946688 00000100:00100000:3.0:1789760798.093298:0:12534:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 483 00000100:00100000:3.0:1789760798.093309:0:12534:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5115500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508946688:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093377:0:12534:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5115500 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508946688:12345-0@lo:400:kworker.0 Request processed in 70us (512us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760798.093386:0:12534:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 483 00000100:00100000:3.0:1789760798.093414:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508947200 00000100:00100000:3.0:1789760798.093418:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508947200 00000100:00100000:3.0:1789760798.093457:0:8100:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508947712 00000100:00100000:3.0:1789760798.093459:0:8100:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508947712 00000100:00100000:3.0:1789760798.093480:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 357 00000100:00100000:3.0:1789760798.093487:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508947200:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093520:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508947200:12345-0@lo:400:kworker.0 Request processed in 34us (588us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760798.093527:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 357 00000100:00100000:3.0:1789760798.093540:0:8100:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 358 00000100:00100000:3.0:1789760798.093546:0:8100:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508947712:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093574:0:8100:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5117800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:lustre-MDT0000-mdtlov_UUID+5:3478:x1876699508947712:12345-0@lo:400:kworker.0 Request processed in 28us (598us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760798.093580:0:8100:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 358 00000100:00100000:3.0:1789760798.093604:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508946304 00000100:00100000:3.0:1789760798.093608:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508946304 00000100:00100000:3.0:1789760798.093635:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508946816 00000100:00100000:3.0:1789760798.093638:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508946816 00000100:00100000:3.0:1789760798.093655:0:8906:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508947328 00000100:00100000:3.0:1789760798.093658:0:8906:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508947328 00000100:00100000:3.0:1789760798.093680:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1025 00000100:00100000:3.0:1789760798.093687:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508946304:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093721:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3478:x1876699508946304:12345-0@lo:400:kworker.0 Request processed in 35us (663us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760798.093728:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1025 00000100:00100000:3.0:1789760798.093740:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1026 00000100:00100000:3.0:1789760798.093747:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508946816:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093774:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3478:x1876699508946816:12345-0@lo:400:kworker.0 Request processed in 29us (640us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760798.093781:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1026 00000100:00100000:3.0:1789760798.093792:0:8906:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1027 00000100:00100000:3.0:1789760798.093798:0:8906:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508947328:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093823:0:8906:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3478:x1876699508947328:12345-0@lo:400:kworker.0 Request processed in 25us (644us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760798.093829:0:8906:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1027 00000100:00080000:3.0:1789760798.093863:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:3.0:1789760798.093869:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:3.0:1789760798.093872:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1789760798.093951:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb9f1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3478:1876699508946304:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093966:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb9f3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3478:1876699508946816:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093976:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3478:1876699508947328:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093987:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb9f2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508946688:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.093999:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508947200:0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.094008:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c4579880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3478:1876699508947712:0@lo:400:kworker.0 00000100:00100000:0.0:1789760798.094346:0:6468:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508946176 00000100:00100000:0.0:1789760798.094354:0:6468:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508946176 00000100:00100000:0.0:1789760798.094409:0:6468:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 500 00000100:00100000:0.0:1789760798.094419:0:6468:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8039050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508946176:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1789760798.094474:0:6468:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8039050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:b1dc06f6-5735-4506-9199-d70c6d2a6cb0+24:3478:x1876699508946176:12345-0@lo:400:kworker.0 Request processed in 58us (1720us total) trans 0 rc 0/0 00000100:00100000:0.0:1789760798.094481:0:6468:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 500 00000100:00100000:1.0:1789760798.094859:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4f5115c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508946048:0@lo:13:osp-pre-1-1.0 00000100:00100000:1.0:1789760798.094923:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760798.094953:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb9f0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508946560:0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.094992:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760798.095010:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508947072:0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095038:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760798.095061:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c4578e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508947584:0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095103:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760798.095128:0:3481:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [2->3] 00000100:00100000:1.0:1789760798.095135:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4fb9f0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508946432:0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095162:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760798.095181:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf4c12ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0000_UUID:3481:1876699508946944:0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095204:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760798.095221:0:3481:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf5018ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0001_UUID:3481:1876699508947456:0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095243:0:3481:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1789760798.095321:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508947072 00000100:00100000:1.0:1789760798.095328:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508947072 00000100:00100000:1.0:1789760798.095378:0:8103:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508947584 00000100:00100000:1.0:1789760798.095382:0:8103:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508947584 00000100:00100000:1.0:1789760798.095410:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 359 00000100:00100000:1.0:1789760798.095419:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b1b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508947072:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095472:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b1b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508947072:12345-0@lo:400:kworker.0 Request processed in 55us (432us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760798.095479:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 359 00000100:00100000:1.0:1789760798.095498:0:8103:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 360 00000100:00100000:1.0:1789760798.095504:0:8103:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508947584:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095534:0:8103:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_002:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508947584:12345-0@lo:400:kworker.0 Request processed in 30us (430us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760798.095540:0:8103:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 360 00000100:00100000:1.0:1789760798.095559:0:6479:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508946432 00000100:00100000:1.0:1789760798.095563:0:6479:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508946432 00000100:00100000:1.0:1789760798.095587:0:6479:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 484 00000100:00100000:1.0:1789760798.095594:0:6479:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3481:x1876699508946432:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095628:0:6479:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0001-mdtlov_UUID+6:3481:x1876699508946432:12345-0@lo:400:kworker.0 Request processed in 34us (465us total) trans 0 rc 0/0 00000100:00100000:1.0:1789760798.095634:0:6479:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 484 00000100:00100000:1.0:1789760798.095672:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb9f0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508946432:0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095701:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508947072:0@lo:400:kworker.0 00000100:00100000:1.0:1789760798.095711:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c4578e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508947584:0@lo:400:kworker.0 00000100:00100000:2.0:1789760798.096975:0:3479:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8bf3c4579500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508947840:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760798.097055:0:3479:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1789760798.097934:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508946048 00000100:00100000:3.0:1789760798.097950:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508946048 00000100:00100000:3.0:1789760798.098121:0:8104:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508947840 00000100:00100000:3.0:1789760798.098125:0:8104:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508947840 00000100:00100000:3.0:1789760798.098145:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 424 00000100:00100000:3.0:1789760798.098155:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508946048:12345-0@lo:13:osp-pre-1-1.0 00002000:00000024:3.0:1789760798.098200:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:3.0:1789760798.098280:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9c15c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3481:x1876699508946048:12345-0@lo:13:osp-pre-1-1.0 Request processed in 126us (3356us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760798.098288:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 424 00000100:00100000:3.0:1789760798.098307:0:8104:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 425 00000100:00100000:3.0:1789760798.098314:0:8104:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df829180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508947840:12345-0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760798.098318:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508946560 00000100:00100000:2.0:1789760798.098324:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508946560 00002000:00000024:3.0:1789760798.098324:0:8104:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 40697856 pending 0 free 3859808256 avail 3857711104 00000100:00100000:2.0:1789760798.098353:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508946944 00000100:00100000:3.0:1789760798.098355:0:8104:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df829180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3479:x1876699508947840:12345-0@lo:13:osp-pre-1-0.0 Request processed in 43us (1301us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760798.098356:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508946944 00000100:00100000:3.0:1789760798.098361:0:8104:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 425 00000100:00100000:2.0:1789760798.098379:0:6473:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699508947456 00000100:00100000:2.0:1789760798.098382:0:6473:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699508947456 00000100:00100000:2.0:1789760798.098403:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1028 00000100:00100000:2.0:1789760798.098409:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508946560:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.098432:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760798.098448:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c9b19c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0001_UUID+5:3481:x1876699508946560:12345-0@lo:400:kworker.0 Request processed in 40us (3456us total) trans 0 rc 0/0 00000100:00100000:2.0:1789760798.098455:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1028 00000100:00100000:3.0:1789760798.098462:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484387456 00000100:00100000:2.0:1789760798.098466:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1029 00000100:00100000:3.0:1789760798.098466:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484387456 00000100:00100000:2.0:1789760798.098472:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0000_UUID+5:3481:x1876699508946944:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1789760798.098501:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f59500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0000_UUID+5:3481:x1876699508946944:12345-0@lo:400:kworker.0 Request processed in 30us (3297us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760798.098505:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787207031570 time 787374057254 state 0x0 ref 1 00000100:00100000:2.0:1789760798.098507:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1029 00000100:00100000:2.0:1789760798.098518:0:6473:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1030 00000100:00100000:3.0:1789760798.098521:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787374317670 time 787374057254 state 0x1 ref 1 00000100:00100000:2.0:1789760798.098524:0:6473:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0001_UUID+5:3481:x1876699508947456:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1789760798.098542:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787374317670 time 787429472814 state 0x0 ref 1 00000100:00100000:2.0:1789760798.098551:0:6473:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0001-lwp-OST0001_UUID+5:3481:x1876699508947456:12345-0@lo:400:kworker.0 Request processed in 28us (3307us total) trans 0 rc 0/0 00000100:00100000:3.0:1789760798.098551:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5114380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484387456:12345-192.168.203.10@tcp:4:dd.0 00000100:00100000:2.0:1789760798.098557:0:6473:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1030 00000100:00080000:2.0:1789760798.098583:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 00000100:00080000:2.0:1789760798.098588:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:2.0:1789760798.098591:0:6469:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00010000:3.0:1789760798.098603:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26099712, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00000100:00100000:2.0:1789760798.098633:0:3478:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb9f0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b1dc06f6-5735-4506-9199-d70c6d2a6cb0:3478:1876699508946176:0@lo:400:kworker.0 00000100:00100000:2.0:1789760798.099128:0:3479:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf3c4579500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3479:1876699508947840:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1789760798.099172:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4c12ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0000_UUID:3481:1876699508946944:0@lo:400:kworker.0 00000100:00100000:2.0:1789760798.099189:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf5018ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-lwp-OST0001_UUID:3481:1876699508947456:0@lo:400:kworker.0 00000100:00100000:2.0:1789760798.099207:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4f5115c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3481:1876699508946048:0@lo:13:osp-pre-1-1.0 00000100:00100000:2.0:1789760798.099221:0:3481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8bf4fb9f0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0001_UUID:3481:1876699508946560:0@lo:400:kworker.0 00010000:00010000:3.0:1789760798.099817:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.099842:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:3.0:1789760798.099848:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760798.099857:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26099712->26103807] (req 26099712->26103807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.099876:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26099712->26103807] (req 26099712->26103807) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760798.141987:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26099712 -> 26103808 00010000:00010000:3.0:1789760798.142016:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26099712->26103807] (req 26099712->26103807) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.142034:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26099712->26103807] (req 26099712->26103807) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.142045:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26099712->26103807] (req 26099712->26103807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.142054:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760798.142060:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26099712->26103807] (req 26099712->26103807) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.142098:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26099712->26103807] (req 26099712->26103807) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.142126:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26099712->26103807] (req 26099712->26103807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.142134:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a7f9 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26099712->26103807] (req 26099712->26103807) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760798.142343:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5114380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484387456:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43783us (43905us total) trans 4294973690 rc 0/0 00000100:00100000:3.0:1789760798.142356:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6401 00000100:00100000:3.0:1789760798.142369:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787374317670 time 787429472814 state 0x2 ref 0 00000001:00080000:2.0:1789760798.143372:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973690 is committed 00000100:00100000:2.0:1789760798.146159:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:0.0:1789760798.146344:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484387584 00000100:00100000:0.0:1789760798.146358:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484387584 00000100:00100000:0.0:1789760798.146442:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787374317670 time 787429472814 state 0x0 ref 1 00000100:00100000:0.0:1789760798.146463:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787429733230 time 787429472814 state 0x1 ref 1 00000100:00100000:0.0:1789760798.146497:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787429733230 time 787477428132 state 0x0 ref 1 00000100:00100000:0.0:1789760798.146512:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484387584:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:0.0:1789760798.146576:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26103808, mtime: 0x6aad951d, atime: 0x0, ctime: 0x6aad951d, blocks: 0xc807 00010000:00010000:0.0:1789760798.146587:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.146604:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:0.0:1789760798.146609:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:0.0:1789760798.146617:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26103808->26107903] (req 26103808->26107903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.146633:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26103808->26107903] (req 26103808->26107903) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760798.185766:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26103808 -> 26107904 00002000:00010000:2.0:1789760798.185779:0:14643:0:(ofd_lvb.c:313:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb mtime from disk: 1789760797 -> 1789760798 00002000:00010000:2.0:1789760798.185782:0:14643:0:(ofd_lvb.c:325:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb ctime from disk: 1789760797 -> 1789760798 00010000:00010000:2.0:1789760798.185800:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26103808->26107903] (req 26103808->26107903) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.185815:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26103808->26107903] (req 26103808->26107903) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.185825:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26103808->26107903] (req 26103808->26107903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.185834:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4cb199a00 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760798.185839:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26103808->26107903] (req 26103808->26107903) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.185849:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26103808->26107903] (req 26103808->26107903) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.185873:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26103808->26107903] (req 26103808->26107903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.185881:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001b3289b7/0x2cf382639620a800 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26103808->26107903] (req 26103808->26107903) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760798.186192:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2424:x1876699484387584:12345-192.168.203.10@tcp:4:dd.0 Request processed in 39675us (40041us total) trans 4294973691 rc 0/0 00000100:00100000:2.0:1789760798.186489:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6402 00000100:00100000:2.0:1789760798.186501:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787429733230 time 787477428132 state 0x2 ref 0 00000001:00080000:2.0:1789760798.186557:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973691 is committed 00000100:00100000:0.0:1789760798.194760:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760798.194900:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484387712 00000100:00100000:2.0:1789760798.194915:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484387712 00000100:00100000:2.0:1789760798.195007:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787429733230 time 787477428132 state 0x0 ref 1 00000100:00100000:2.0:1789760798.195028:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787477688548 time 787477428132 state 0x1 ref 1 00000100:00100000:2.0:1789760798.195108:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787477688548 time 787525994685 state 0x0 ref 1 00000100:00100000:2.0:1789760798.195125:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484387712:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760798.195206:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26107904, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:2.0:1789760798.195218:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.195238:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:2.0:1789760798.195243:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760798.195251:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26107904->26111999] (req 26107904->26111999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.195271:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26107904->26111999] (req 26107904->26111999) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760798.249312:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973692 is committed 00002000:00010000:0.0:1789760798.249398:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26107904 -> 26112000 00010000:00010000:0.0:1789760798.249496:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26107904->26111999] (req 26107904->26111999) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.249509:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26107904->26111999] (req 26107904->26111999) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.249516:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26107904->26111999] (req 26107904->26111999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.249526:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:0.0:1789760798.249530:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26107904->26111999] (req 26107904->26111999) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.249537:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26107904->26111999] (req 26107904->26111999) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.249557:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26107904->26111999] (req 26107904->26111999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.249562:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a807 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26107904->26111999] (req 26107904->26111999) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789760798.250435:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2424:x1876699484387712:12345-192.168.203.10@tcp:4:dd.0 Request processed in 55301us (55679us total) trans 4294973692 rc 0/0 00000100:00100000:0.0:1789760798.250450:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6403 00000100:00100000:0.0:1789760798.250464:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787477688548 time 787525994685 state 0x2 ref 0 00000100:00100000:0.0:1789760798.253341:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760798.254125:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484387840 00000100:00100000:3.0:1789760798.254141:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484387840 00000100:00100000:3.0:1789760798.254329:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787477688548 time 787525994685 state 0x0 ref 1 00000100:00100000:3.0:1789760798.254350:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787526255101 time 787525994685 state 0x1 ref 1 00000100:00100000:3.0:1789760798.254389:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787526255101 time 787585319968 state 0x0 ref 1 00000100:00100000:3.0:1789760798.254477:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484387840:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760798.254549:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26112000, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:3.0:1789760798.254560:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.254583:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:3.0:1789760798.254588:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760798.254596:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26112000->26116095] (req 26112000->26116095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.254613:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26112000->26116095] (req 26112000->26116095) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760798.303392:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26112000 -> 26116096 00010000:00010000:2.0:1789760798.303434:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26112000->26116095] (req 26112000->26116095) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.303451:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26112000->26116095] (req 26112000->26116095) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.303460:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26112000->26116095] (req 26112000->26116095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.303474:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760798.303480:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26112000->26116095] (req 26112000->26116095) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.303491:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26112000->26116095] (req 26112000->26116095) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.303513:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26112000->26116095] (req 26112000->26116095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.303521:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a80e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26112000->26116095] (req 26112000->26116095) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760798.303659:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484387840:12345-192.168.203.10@tcp:4:dd.0 Request processed in 49250us (50324us total) trans 4294973693 rc 0/0 00000100:00100000:2.0:1789760798.303671:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6404 00000100:00100000:2.0:1789760798.303687:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787526255101 time 787585319968 state 0x2 ref 0 00000001:00080000:2.0:1789760798.304181:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973693 is committed 00000100:00100000:2.0:1789760798.310195:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760798.310337:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484387968 00000100:00100000:3.0:1789760798.310349:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484387968 00000100:00100000:3.0:1789760798.310467:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787526255101 time 787585319968 state 0x0 ref 1 00000100:00100000:3.0:1789760798.310483:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787585580384 time 787585319968 state 0x1 ref 1 00000100:00100000:3.0:1789760798.310510:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787585580384 time 787641440410 state 0x0 ref 1 00000100:00100000:3.0:1789760798.310525:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484387968:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760798.310597:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26116096, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:3.0:1789760798.310609:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.310629:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:3.0:1789760798.310633:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760798.310641:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26116096->26120191] (req 26116096->26120191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.310658:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26116096->26120191] (req 26116096->26120191) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:2.0:1789760798.352816:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26116096 -> 26120192 00010000:00010000:2.0:1789760798.352841:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26116096->26120191] (req 26116096->26120191) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.352853:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26116096->26120191] (req 26116096->26120191) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.352860:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26116096->26120191] (req 26116096->26120191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.352865:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760798.352869:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26116096->26120191] (req 26116096->26120191) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.352876:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26116096->26120191] (req 26116096->26120191) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.352905:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26116096->26120191] (req 26116096->26120191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.352915:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a815 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26116096->26120191] (req 26116096->26120191) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760798.353537:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973694 is committed 00000100:00100000:2.0:1789760798.353572:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484387968:12345-192.168.203.10@tcp:4:dd.0 Request processed in 43042us (43385us total) trans 4294973694 rc 0/0 00000100:00100000:2.0:1789760798.353586:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6405 00000100:00100000:2.0:1789760798.353600:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787585580384 time 787641440410 state 0x2 ref 0 00000100:00100000:0.0:1789760798.356123:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760798.356210:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484388096 00000100:00100000:2.0:1789760798.356227:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484388096 00000100:00100000:2.0:1789760798.356304:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787585580384 time 787641440410 state 0x0 ref 1 00000100:00100000:2.0:1789760798.356323:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787641700826 time 787641440410 state 0x1 ref 1 00000100:00100000:2.0:1789760798.356359:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787641700826 time 787687290027 state 0x0 ref 1 00000100:00100000:2.0:1789760798.356375:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484388096:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760798.356440:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26120192, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:2.0:1789760798.356520:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.356540:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:2.0:1789760798.356545:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760798.356550:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26120192->26124287] (req 26120192->26124287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.356569:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26120192->26124287] (req 26120192->26124287) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760798.397303:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973695 is committed 00002000:00010000:2.0:1789760798.398153:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26120192 -> 26124288 00010000:00010000:2.0:1789760798.398177:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26120192->26124287] (req 26120192->26124287) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.398193:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26120192->26124287] (req 26120192->26124287) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.398204:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26120192->26124287] (req 26120192->26124287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.398213:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a79200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760798.398219:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26120192->26124287] (req 26120192->26124287) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.398228:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26120192->26124287] (req 26120192->26124287) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.398253:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26120192->26124287] (req 26120192->26124287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.398263:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000b2720d3f/0x2cf382639620a81c lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26120192->26124287] (req 26120192->26124287) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760798.398505:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4578e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484388096:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42126us (42390us total) trans 4294973695 rc 0/0 00000100:00100000:2.0:1789760798.398518:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6406 00000100:00100000:2.0:1789760798.398530:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787641700826 time 787687290027 state 0x2 ref 0 00000100:00100000:3.0:1789760798.401705:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760798.401772:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484388224 00000100:00100000:2.0:1789760798.401784:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484388224 00000100:00100000:2.0:1789760798.401854:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787641700826 time 787687290027 state 0x0 ref 1 00000100:00100000:2.0:1789760798.401873:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787687550443 time 787687290027 state 0x1 ref 1 00000100:00100000:2.0:1789760798.401918:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787687550443 time 787732847388 state 0x0 ref 1 00000100:00100000:2.0:1789760798.401932:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484388224:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760798.402006:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26124288, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:2.0:1789760798.402016:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.402035:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:2.0:1789760798.402039:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760798.402047:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26124288->26128383] (req 26124288->26128383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.402089:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26124288->26128383] (req 26124288->26128383) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760798.444195:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26124288 -> 26128384 00010000:00010000:3.0:1789760798.444224:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26124288->26128383] (req 26124288->26128383) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.444240:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26124288->26128383] (req 26124288->26128383) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.444251:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26124288->26128383] (req 26124288->26128383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.444260:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760798.444266:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26124288->26128383] (req 26124288->26128383) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.444276:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26124288->26128383] (req 26124288->26128383) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.444298:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26124288->26128383] (req 26124288->26128383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.444305:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000054f9383e/0x2cf382639620a823 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26124288->26128383] (req 26124288->26128383) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760798.444500:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484388224:12345-192.168.203.10@tcp:4:dd.0 Request processed in 42564us (42802us total) trans 4294973696 rc 0/0 00000100:00100000:3.0:1789760798.444513:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6407 00000100:00100000:3.0:1789760798.444525:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787687550443 time 787732847388 state 0x2 ref 0 00000100:00100000:3.0:1789760798.447453:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760798.447626:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484388352 00000100:00100000:2.0:1789760798.447639:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484388352 00000100:00100000:2.0:1789760798.447726:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787687550443 time 787732847388 state 0x0 ref 1 00000100:00100000:2.0:1789760798.447745:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787733107804 time 787732847388 state 0x1 ref 1 00000100:00100000:2.0:1789760798.447783:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787733107804 time 787778713873 state 0x0 ref 1 00000100:00100000:2.0:1789760798.447799:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f5116a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484388352:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760798.447868:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26128384, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:2.0:1789760798.447879:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.447910:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:2.0:1789760798.447916:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760798.447924:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26128384->26132479] (req 26128384->26132479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.447941:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26128384->26132479] (req 26128384->26132479) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760798.450837:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973696 is committed 00000001:00080000:0.0:1789760798.481862:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973697 is committed 00002000:00010000:2.0:1789760798.482022:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26128384 -> 26132480 00010000:00010000:2.0:1789760798.482041:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26128384->26132479] (req 26128384->26132479) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.482104:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26128384->26132479] (req 26128384->26132479) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.482117:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26128384->26132479] (req 26128384->26132479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.482126:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760798.482132:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26128384->26132479] (req 26128384->26132479) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.482143:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26128384->26132479] (req 26128384->26132479) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.482169:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26128384->26132479] (req 26128384->26132479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.482177:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a82a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26128384->26132479] (req 26128384->26132479) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760798.482309:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f5116a00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484388352:12345-192.168.203.10@tcp:4:dd.0 Request processed in 34494us (34849us total) trans 4294973697 rc 0/0 00000100:00100000:2.0:1789760798.482321:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6408 00000100:00100000:2.0:1789760798.482333:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787733107804 time 787778713873 state 0x2 ref 0 00000100:00100000:2.0:1789760798.489290:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760798.489406:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484388480 00000100:00100000:1.0:1789760798.489420:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484388480 00000100:00100000:1.0:1789760798.489498:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787733107804 time 787778713873 state 0x0 ref 1 00000100:00100000:1.0:1789760798.489518:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787778974289 time 787778713873 state 0x1 ref 1 00000100:00100000:1.0:1789760798.489553:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787778974289 time 787820483125 state 0x0 ref 1 00000100:00100000:1.0:1789760798.489568:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4f7068c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484388480:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760798.489637:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26132480, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:1.0:1789760798.489647:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.489668:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:1.0:1789760798.489673:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760798.489681:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26132480->26136575] (req 26132480->26136575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.489697:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26132480->26136575] (req 26132480->26136575) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760798.529941:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973698 is committed 00002000:00010000:2.0:1789760798.530760:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26132480 -> 26136576 00010000:00010000:2.0:1789760798.530781:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26132480->26136575] (req 26132480->26136575) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.530797:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26132480->26136575] (req 26132480->26136575) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.530807:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26132480->26136575] (req 26132480->26136575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.530816:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4ce000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:2.0:1789760798.530821:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26132480->26136575] (req 26132480->26136575) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.530831:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26132480->26136575] (req 26132480->26136575) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.530856:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26132480->26136575] (req 26132480->26136575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.530865:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000001ffadc89/0x2cf382639620a831 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26132480->26136575] (req 26132480->26136575) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789760798.531149:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4f7068c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484388480:12345-192.168.203.10@tcp:4:dd.0 Request processed in 41576us (41867us total) trans 4294973698 rc 0/0 00000100:00100000:2.0:1789760798.531163:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6409 00000100:00100000:2.0:1789760798.531176:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787778974289 time 787820483125 state 0x2 ref 0 00000100:00100000:0.0:1789760798.534625:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760798.534766:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484388608 00000100:00100000:3.0:1789760798.534779:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484388608 00000100:00100000:3.0:1789760798.534850:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787778974289 time 787820483125 state 0x0 ref 1 00000100:00100000:3.0:1789760798.534870:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787820743541 time 787820483125 state 0x1 ref 1 00000100:00100000:3.0:1789760798.534944:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787820743541 time 787865873899 state 0x0 ref 1 00000100:00100000:3.0:1789760798.534958:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484388608:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760798.535033:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26136576, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:3.0:1789760798.535045:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.535089:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:3.0:1789760798.535095:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760798.535103:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26136576->26140671] (req 26136576->26140671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.535119:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26136576->26140671] (req 26136576->26140671) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760798.565947:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973699 is committed 00002000:00010000:3.0:1789760798.566853:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26136576 -> 26140672 00010000:00010000:3.0:1789760798.566876:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26136576->26140671] (req 26136576->26140671) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.566899:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26136576->26140671] (req 26136576->26140671) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.566914:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26136576->26140671] (req 26136576->26140671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.566920:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760798.566925:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26136576->26140671] (req 26136576->26140671) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.566933:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26136576->26140671] (req 26136576->26140671) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.566952:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26136576->26140671] (req 26136576->26140671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.566959:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a838 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26136576->26140671] (req 26136576->26140671) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760798.567059:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484388608:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32098us (32443us total) trans 4294973699 rc 0/0 00000100:00100000:3.0:1789760798.567107:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6410 00000100:00100000:3.0:1789760798.567116:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787820743541 time 787865873899 state 0x2 ref 0 00000100:00100000:0.0:1789760798.572884:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760798.572993:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484388736 00000100:00100000:3.0:1789760798.573008:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484388736 00000100:00100000:3.0:1789760798.573097:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787820743541 time 787865873899 state 0x0 ref 1 00000100:00100000:3.0:1789760798.573115:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787866134315 time 787865873899 state 0x1 ref 1 00000100:00100000:3.0:1789760798.573144:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787866134315 time 787904075543 state 0x0 ref 1 00000100:00100000:3.0:1789760798.573158:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484388736:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760798.573207:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26140672, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:3.0:1789760798.573217:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.573234:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:3.0:1789760798.573239:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760798.573246:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26140672->26144767] (req 26140672->26144767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.573262:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26140672->26144767] (req 26140672->26144767) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789760798.606498:0:8117:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00080000:3.0:1789760798.607808:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973700 is committed 00002000:00010000:1.0:1789760798.607925:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26140672 -> 26144768 00010000:00010000:1.0:1789760798.607944:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26140672->26144767] (req 26140672->26144767) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.607959:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26140672->26144767] (req 26140672->26144767) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.607968:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26140672->26144767] (req 26140672->26144767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.608051:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cc000 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760798.608057:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26140672->26144767] (req 26140672->26144767) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.608089:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26140672->26144767] (req 26140672->26144767) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.608114:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26140672->26144767] (req 26140672->26144767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.608122:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000053e01392/0x2cf382639620a83f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26140672->26144767] (req 26140672->26144767) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760798.608288:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c457bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484388736:12345-192.168.203.10@tcp:4:dd.0 Request processed in 35125us (35409us total) trans 4294973700 rc 0/0 00000100:00100000:1.0:1789760798.608300:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6411 00000100:00100000:1.0:1789760798.608313:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787866134315 time 787904075543 state 0x2 ref 0 00000100:00100000:0.0:1789760798.611269:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760798.611381:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484388864 00000100:00100000:2.0:1789760798.611395:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484388864 00000100:00100000:2.0:1789760798.611465:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787866134315 time 787904075543 state 0x0 ref 1 00000100:00100000:2.0:1789760798.611485:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787904335959 time 787904075543 state 0x1 ref 1 00000100:00100000:2.0:1789760798.611565:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787904335959 time 787942494933 state 0x0 ref 1 00000100:00100000:2.0:1789760798.611579:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2425:x1876699484388864:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760798.611647:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26144768, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:2.0:1789760798.611658:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.611679:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:2.0:1789760798.611684:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760798.611692:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26144768->26148863] (req 26144768->26148863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.611709:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26144768->26148863] (req 26144768->26148863) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760798.643697:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26144768 -> 26148864 00010000:00010000:3.0:1789760798.643723:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26144768->26148863] (req 26144768->26148863) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.643737:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26144768->26148863] (req 26144768->26148863) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.643747:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26144768->26148863] (req 26144768->26148863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.643760:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760798.643766:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26144768->26148863] (req 26144768->26148863) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.643778:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26144768->26148863] (req 26144768->26148863) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.643804:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26144768->26148863] (req 26144768->26148863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.643812:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000c486a791/0x2cf382639620a846 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26144768->26148863] (req 26144768->26148863) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760798.643951:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf3c4579500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2425:x1876699484388864:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32366us (32688us total) trans 4294973701 rc 0/0 00000100:00100000:3.0:1789760798.643964:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6412 00000100:00100000:3.0:1789760798.644120:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787904335959 time 787942494933 state 0x2 ref 0 00000001:00080000:1.0:1789760798.644222:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973701 is committed 00000100:00100000:2.0:1789760798.646786:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760798.646885:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484388992 00000100:00100000:1.0:1789760798.646907:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484388992 00000100:00100000:1.0:1789760798.646988:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787904335959 time 787942494933 state 0x0 ref 1 00000100:00100000:1.0:1789760798.647008:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787942755349 time 787942494933 state 0x1 ref 1 00000100:00100000:1.0:1789760798.647042:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787942755349 time 787977973356 state 0x0 ref 1 00000100:00100000:1.0:1789760798.647056:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484388992:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760798.647153:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26148864, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:1.0:1789760798.647163:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.647184:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=787s v=(0 0) 00010000:00010000:1.0:1789760798.647189:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760798.647196:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26148864->26152959] (req 26148864->26152959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.647212:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26148864->26152959] (req 26148864->26152959) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760798.678472:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26148864 -> 26152960 00010000:00010000:3.0:1789760798.678500:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26148864->26152959] (req 26148864->26152959) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.678517:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26148864->26152959] (req 26148864->26152959) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.678534:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26148864->26152959] (req 26148864->26152959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.678543:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760798.678548:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26148864->26152959] (req 26148864->26152959) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.678558:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26148864->26152959] (req 26148864->26152959) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.678586:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26148864->26152959] (req 26148864->26152959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.678594:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000075e5e55f/0x2cf382639620a84d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26148864->26152959] (req 26148864->26152959) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760798.678722:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484388992:12345-192.168.203.10@tcp:4:dd.0 Request processed in 31662us (31942us total) trans 4294973702 rc 0/0 00000100:00100000:3.0:1789760798.678734:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6413 00000100:00100000:3.0:1789760798.678746:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787942755349 time 787977973356 state 0x2 ref 0 00000001:00080000:1.0:1789760798.679770:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973702 is committed 00000100:00100000:2.0:1789760798.681930:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760798.682098:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484389120 00000100:00100000:3.0:1789760798.682114:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484389120 00000100:00100000:3.0:1789760798.682190:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787942755349 time 787977973356 state 0x0 ref 1 00000100:00100000:3.0:1789760798.682257:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787978233772 time 787977973356 state 0x1 ref 1 00000100:00100000:3.0:1789760798.682325:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787978233772 time 788013254715 state 0x0 ref 1 00000100:00100000:3.0:1789760798.682339:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df828000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484389120:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760798.682401:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26152960, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:3.0:1789760798.682411:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.682430:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=788s v=(0 0) 00010000:00010000:3.0:1789760798.682435:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760798.682442:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26152960->26157055] (req 26152960->26157055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.682458:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26152960->26157055] (req 26152960->26157055) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00002000:00010000:3.0:1789760798.714684:0:14643:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26152960 -> 26157056 00010000:00010000:3.0:1789760798.714714:0:14643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26152960->26157055] (req 26152960->26157055) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.714730:0:14643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26152960->26157055] (req 26152960->26157055) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.714741:0:14643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26152960->26157055] (req 26152960->26157055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.714750:0:14643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760798.714756:0:14643:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26152960->26157055] (req 26152960->26157055) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.714768:0:14643:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26152960->26157055] (req 26152960->26157055) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.714793:0:14643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26152960->26157055] (req 26152960->26157055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.714801:0:14643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a854 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26152960->26157055] (req 26152960->26157055) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760798.714934:0:14643:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df828000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+7:2422:x1876699484389120:12345-192.168.203.10@tcp:4:dd.0 Request processed in 32590us (33012us total) trans 4294973703 rc 0/0 00000100:00100000:3.0:1789760798.714947:0:14643:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6414 00000100:00100000:3.0:1789760798.714959:0:14643:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787978233772 time 788013254715 state 0x2 ref 0 00000001:00080000:2.0:1789760798.715779:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973703 is committed 00000100:00100000:2.0:1789760798.725039:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:1.0:1789760798.727056:0:8108:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484389248 00000100:00100000:1.0:1789760798.727141:0:8108:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484389248 00000100:00100000:1.0:1789760798.727254:0:8108:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 787978233772 time 788013254715 state 0x0 ref 1 00000100:00100000:1.0:1789760798.727274:0:8108:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788013515131 time 788013254715 state 0x1 ref 1 00000100:00100000:1.0:1789760798.727315:0:8108:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788013515131 time 788058245449 state 0x0 ref 1 00000100:00100000:1.0:1789760798.727331:0:8108:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484389248:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:1.0:1789760798.727399:0:8108:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26157056, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:1.0:1789760798.727411:0:8108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.727431:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=788s v=(0 0) 00010000:00010000:1.0:1789760798.727436:0:8108:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:1.0:1789760798.727443:0:8108:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26157056->26161151] (req 26157056->26161151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.727459:0:8108:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26157056->26161151] (req 26157056->26161151) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:3.0:1789760798.771688:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973704 is committed 00002000:00010000:1.0:1789760798.771864:0:8108:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26157056 -> 26161152 00010000:00010000:1.0:1789760798.771885:0:8108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26157056->26161151] (req 26157056->26161151) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.771911:0:8108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26157056->26161151] (req 26157056->26161151) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.771921:0:8108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26157056->26161151] (req 26157056->26161151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.771929:0:8108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ff4cf200 already unused, calling callback (ffffffffc1781350) 00010000:00010000:1.0:1789760798.771934:0:8108:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26157056->26161151] (req 26157056->26161151) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.771945:0:8108:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26157056->26161151] (req 26157056->26161151) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.771971:0:8108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26157056->26161151] (req 26157056->26161151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789760798.772042:0:8108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 0000000083aff141/0x2cf382639620a85b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26157056->26161151] (req 26157056->26161151) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789760798.772205:0:8108:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82bb80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484389248:12345-192.168.203.10@tcp:4:dd.0 Request processed in 44870us (47176us total) trans 4294973704 rc 0/0 00000100:00100000:1.0:1789760798.772217:0:8108:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6415 00000100:00100000:1.0:1789760798.772230:0:8108:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788013515131 time 788058245449 state 0x2 ref 0 00000100:00100000:1.0:1789760798.775521:0:3473:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:2.0:1789760798.775679:0:8107:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484389376 00000100:00100000:2.0:1789760798.775750:0:8107:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484389376 00000100:00100000:2.0:1789760798.775953:0:8107:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788013515131 time 788058245449 state 0x0 ref 1 00000100:00100000:2.0:1789760798.775971:0:8107:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788058505865 time 788058245449 state 0x1 ref 1 00000100:00100000:2.0:1789760798.776320:0:8107:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788058505865 time 788107248907 state 0x0 ref 1 00000100:00100000:2.0:1789760798.776338:0:8107:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484389376:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:2.0:1789760798.776404:0:8107:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26161152, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:2.0:1789760798.776415:0:8107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.776434:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=788s v=(0 0) 00010000:00010000:2.0:1789760798.776439:0:8107:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:2.0:1789760798.776446:0:8107:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26161152->26165247] (req 26161152->26165247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789760798.776462:0:8107:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26161152->26165247] (req 26161152->26165247) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:1.0:1789760798.809657:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973705 is committed 00002000:00010000:3.0:1789760798.809853:0:8107:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26161152 -> 26165248 00010000:00010000:3.0:1789760798.809873:0:8107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26161152->26165247] (req 26161152->26165247) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.809898:0:8107:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26161152->26165247] (req 26161152->26165247) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.809910:0:8107:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26161152->26165247] (req 26161152->26165247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.809932:0:8107:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4c8a78600 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760798.809938:0:8107:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26161152->26165247] (req 26161152->26165247) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.809949:0:8107:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26161152->26165247] (req 26161152->26165247) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.809974:0:8107:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26161152->26165247] (req 26161152->26165247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.809983:0:8107:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 000000005bedb9f2/0x2cf382639620a862 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26161152->26165247] (req 26161152->26165247) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8107 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760798.810147:0:8107:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4c8f5b100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_001:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2423:x1876699484389376:12345-192.168.203.10@tcp:4:dd.0 Request processed in 33807us (34634us total) trans 4294973705 rc 0/0 00000100:00100000:3.0:1789760798.810159:0:8107:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6416 00000100:00100000:3.0:1789760798.810172:0:8107:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788058505865 time 788107248907 state 0x2 ref 0 00000100:00100000:2.0:1789760798.813721:0:3472:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760798.813811:0:8106:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484389504 00000100:00100000:3.0:1789760798.813824:0:8106:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484389504 00000100:00100000:3.0:1789760798.813907:0:8106:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788058505865 time 788107248907 state 0x0 ref 1 00000100:00100000:3.0:1789760798.813928:0:8106:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788107509323 time 788107248907 state 0x1 ref 1 00000100:00100000:3.0:1789760798.813965:0:8106:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788107509323 time 788144895823 state 0x0 ref 1 00000100:00100000:3.0:1789760798.813993:0:8106:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484389504:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760798.814061:0:8106:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26165248, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:3.0:1789760798.814111:0:8106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.814131:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=788s v=(0 0) 00010000:00010000:3.0:1789760798.814136:0:8106:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760798.814145:0:8106:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26165248->26169343] (req 26165248->26169343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.814161:0:8106:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26165248->26169343] (req 26165248->26169343) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00080000:0.0:1789760798.855148:0:8131:0:(tgt_lastrcvd.c:983:tgt_cb_last_committed()) lustre-OST0000: transno 4294973706 is committed 00002000:00010000:3.0:1789760798.858929:0:8106:0:(ofd_lvb.c:306:ofd_lvbo_update()) res: [0x280000401:0x2:0x0] updating lvb size from disk: 26165248 -> 26169344 00010000:00010000:3.0:1789760798.858955:0:8106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26165248->26169343] (req 26165248->26169343) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.858971:0:8106:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26165248->26169343] (req 26165248->26169343) gid 0 flags: 0x50010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.859039:0:8106:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26165248->26169343] (req 26165248->26169343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.859048:0:8106:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8bf4ca2cd400 already unused, calling callback (ffffffffc1781350) 00010000:00010000:3.0:1789760798.859053:0:8106:0:(ldlm_request.c:378:ldlm_blocking_ast_nocheck()) ### already unused, calling ldlm_cli_cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26165248->26169343] (req 26165248->26169343) gid 0 flags: 0x40010400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.859081:0:8106:0:(ldlm_request.c:1416:ldlm_cli_cancel_local()) ### server-side local cancel ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26165248->26169343] (req 26165248->26169343) gid 0 flags: 0x40018400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.859107:0:8106:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26165248->26169343] (req 26165248->26169343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.859115:0:8106:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a869 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26165248->26169343] (req 26165248->26169343) gid 0 flags: 0x44819400000000 nid: local remote: 0x0 expref: -99 pid: 8106 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789760798.859302:0:8106:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8bf4df82b800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_000:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484389504:12345-192.168.203.10@tcp:4:dd.0 Request processed in 45305us (45587us total) trans 4294973706 rc 0/0 00000100:00100000:3.0:1789760798.859314:0:8106:0:(nrs_tbf.c:3377:nrs_tbf_req_stop()) NRS stop tbf request from 12345-192.168.203.10@tcp, seq: 6417 00000100:00100000:3.0:1789760798.859326:0:8106:0:(nrs_tbf.c:1542:nrs_tbf_lru_add()) ost_io.0 NRS: TBF LRU add: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788107509323 time 788144895823 state 0x2 ref 0 00000100:00100000:1.0:1789760798.869213:0:3471:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.10@tcp (source: 12345-192.168.203.10@tcp) 00000100:00100000:3.0:1789760798.869546:0:14643:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1876699484389632 00000100:00100000:3.0:1789760798.869559:0:14643:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1876699484389632 00000100:00100000:3.0:1789760798.869640:0:14643:0:(nrs_tbf.c:1562:nrs_tbf_lru_tryhit()) ost_io.0 NRS: TBF LRU hit: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788107509323 time 788144895823 state 0x0 ref 1 00000100:00100000:3.0:1789760798.869659:0:14643:0:(nrs_tbf.c:3322:nrs_tbf_req_add()) ost_io.0 NRS: TBF enqueues: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788145156239 time 788144895823 state 0x1 ref 1 00000100:00100000:3.0:1789760798.869754:0:14643:0:(nrs_tbf.c:3136:nrs_req_dequeue_bps()) ost_io.0 NRS: TBF dequeues in BPS mode: class@ffff8bf4fd72a600 key nid={192.168.203.10@tcp}&opcode={ost_write} rule@0xffff8bf4c81dee00 gen 0 rate 15 depth 3 token 3 deadline 788145156239 time 788200683430 state 0x0 ref 1 00000100:00100000:3.0:1789760798.869769:0:14643:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8bf3c9473480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_003:32f439dd-5b05-4ca1-a11b-dd35989c33cd+5:2422:x1876699484389632:12345-192.168.203.10@tcp:4:dd.0 00002000:00010000:3.0:1789760798.869835:0:14643:0:(ofd_lvb.c:151:ofd_lvbo_init()) res: [0x280000401:0x2:0x0] initial LVB size: 26169344, mtime: 0x6aad951e, atime: 0x0, ctime: 0x6aad951e, blocks: 0xc807 00010000:00010000:3.0:1789760798.869846:0:14643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a870 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.869866:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=788s v=(0 0) 00010000:00010000:3.0:1789760798.869871:0:14643:0:(ldlm_extent.c:408:ldlm_update_contention()) add 0 to [0x280000401:0x2:0x0].0x0 since=0s v=(0 0) 00010000:00010000:3.0:1789760798.869880:0:14643:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a870 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26169344->26173439] (req 26169344->26173439) gid 0 flags: 0x50010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789760798.869907:0:14643:0:(ldlm_request.c:557:ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: filter-lustre-OST0000_UUID lock: 00000000cf95f37b/0x2cf382639620a870 lrc: 3/0,1 mode: PW/PW res: [0x280000401:0x2:0x0].0x0 rrc: 2 type: EXT [26169344->26173439] (req 26169344->26173439) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 14643 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Debug log: 57421 lines, 57421 kept, 0 dropped, 0 bad.